Repository navigation
fix(corpus): bound per-document indexing cost - #1463
Conversation
The corpus indexer's per-row isolation catches throws, but a document whose synchronous cost is unbounded never throws: it starves the daemon's event loop, trips the starvation watchdog, and restart-loops because the same row is re-picked on every start. Bound every unbounded step so a pathological document becomes a catchable, audited failure instead: - zstd (de)compression on the corpus read/write paths moves to Bun's async variants (thread pool, not the event loop), and decompression enforces a decompressed-size ceiling before the string reaches JSON.parse or the chunker. - The chunker rejects block counts and fallback-text sizes past structural ceilings, checked before any linear walk, and clamps heading-ancestry depth so malformed levels cannot make path materialization quadratic. Ceilinged documents land in the existing failed-index-job audit trail with the row id, so a poison document names itself instead of taking the worker down.
|
Warning Review limit reached
Next review available in: 20 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughCorpus storage now compresses asynchronously and bounds decompression at 128 MiB. Compression tests cover compatibility and overflow. Corpus chunking now rejects oversized inputs and limits heading ancestry depth. ChangesCorpus budget hardening
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant CorpusStorage
participant Zstd
participant ObjectStorage
CorpusStorage->>Zstd: Compress payload asynchronously
Zstd-->>CorpusStorage: Compressed bytes
CorpusStorage->>ObjectStorage: Write compressed object
ObjectStorage-->>CorpusStorage: Read compressed object
CorpusStorage->>Zstd: Decompress with byte ceiling
Zstd-->>CorpusStorage: Text or budget error
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
bug
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e505fd3e1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| data: Uint8Array, | ||
| maxBytes: number, | ||
| ): Promise<string> => { | ||
| const out = await Bun.zstdDecompress(data); |
There was a problem hiding this comment.
When a corrupt or hostile zstd frame expands past the available heap, this await asks Bun to decode and allocate the entire output before the following byteLength check runs. The process can therefore OOM and restart onto the same pending row without ever throwing PayloadBudgetError, so this does not provide the promised decompression ceiling; use an output-limited or streaming decoder that enforces the limit while decoding.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed: decompression now goes through node:zlib's zstd with maxOutputLength, so the ceiling is enforced during decoding and the oversized output is never materialized.
| "corpus-write-text", | ||
| async () => | ||
| await s3.write(keys.textKey, zstdCompress(text ?? ""), { | ||
| await s3.write(keys.textKey, await zstdCompressAsync(text ?? ""), { |
There was a problem hiding this comment.
When canonical ingestion receives text, serialized sections, or a serialized AST larger than 128 MiB, these writes still succeed because the writer never applies PAYLOAD_MAX_BYTES. The object-storage branch then persists these keys while setting all PostgreSQL payload columns to null, but every changed reader rejects the objects, leaving the decision unreadable and its index job retrying indefinitely; validate the UTF-8 byte sizes before writing so ingestion takes its existing write-failed path.
AGENTS.md reference: apps/api/AGENTS.md:L23-L23
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Deliberately out of scope here: rejecting at write time interacts with the adapter's halted-cursor retry semantics (a deterministic write rejection would hold the cursor forever), so the write-side ceiling needs the retry-policy change alongside it. Tracked as a follow-up with the attempt-capped quarantine.
There was a problem hiding this comment.
💡 Codex Review
stella/apps/api/src/lib/corpus-index/chunking.ts
Lines 373 to 376 in 413d261
Apply the character ceiling to AST-backed documents
When a usable AST has at most 400,000 blocks but their combined plainText exceeds 30,000,000 characters, this branch calls chunkBlocks and returns without ever reaching the only MAX_CHUNK_INPUT_CHARS check in chunkPlainText. A single large block therefore bypasses the intended per-document ceiling, builds and serializes an oversized NDJSON row, and can monopolize the daemon or exceed the corpus-index request budget; count aggregate AST text before chunking as well.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (ast !== null && ast.blocks.length > MAX_CHUNK_BLOCKS) { | ||
| throw new ChunkBudgetError({ | ||
| message: `Document has ${ast.blocks.length} blocks; ceiling is ${MAX_CHUNK_BLOCKS}`, | ||
| }); |
There was a problem hiding this comment.
When a stored AST exceeds 400,000 blocks, this new throw is deterministic, but the shared indexer catches it as a per-row read failure without changing the row's pending state; selectMissing can select it again, and build-corpus-index.ts treats a batch that indexed zero rows as completion. Enough over-budget rows can therefore fill the selection and stop a rebuild while healthy rows remain pending, with later runs appending the same failure audits. Quarantine or terminally mark budget failures, or reject them before persistence. This is distinct fresh evidence from the prior 128-MiB mismatch: the lower block ceiling triggers on otherwise readable objects.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed, and stated as the known follow-up in the PR description: budget failures are cheap and audited now, and the attempt-capped quarantine that stops them re-occupying batch slots lands separately.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/api/src/lib/compression.test.ts`:
- Around line 27-33: Add await to the rejects assertion in the “a payload past
the decompressed ceiling throws instead of returning” test so Bun waits for the
asynchronous rejection matcher to complete. Keep the existing
zstdDecompressToStringBounded call and ceiling error expectation unchanged.
In `@apps/api/src/lib/compression.ts`:
- Around line 41-52: Update zstdDecompressToStringBounded to use a streaming
bounded zstd decompression path, such as node:zlib createZstdDecompress, rather
than Bun.zstdDecompress. Track decoded bytes during streaming, abort and raise
PayloadBudgetError as soon as maxBytes is exceeded, and return the UTF-8 result
only after successful completion.
In `@apps/api/src/lib/corpus-index/chunking.ts`:
- Around line 284-288: Update the AST chunking path around the blocks.length
guard to preflight the aggregate length of every block’s plainText and reject
documents exceeding the existing 30M-character budget before constructing
chunks. Preserve the current block-count validation, and add a test covering a
single plainText block that exceeds the text-size limit.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: f417d4f5-b5b3-4ca0-947d-473b6166dd0a
📒 Files selected for processing (6)
apps/api/src/handlers/case-law/corpus-storage.tsapps/api/src/lib/compression.test.tsapps/api/src/lib/compression.tsapps/api/src/lib/corpus-index/chunking.test.tsapps/api/src/lib/corpus-index/chunking.tsapps/api/src/lib/limits.ts
📜 Review details
🧰 Additional context used
📓 Path-based instructions (8)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
In TypeScript, use strict mode, prefer
typeoverinterface, avoidany, and avoid non-null assertions.
**/*.{ts,tsx}: Prefer explicit propagation of backend discriminators through URLs, props, and the full stack rather than hardcoded frontend defaults.
Prefer TypeScript types that make invalid states structurally impossible, including branded types, discriminated unions, and exhaustive checks.
Use named discriminators or domain types for states that may grow; avoid permanent boolean state flags.
Fail fast: validate at boundaries and return or throw early; minimize brace nesting with inverted conditions and early returns.
Use named constants instead of string literals for domain values.
Do not assign directly todocument.cookie.
Avoid spread in loop accumulators; use.push()instead.
Useas constobjects or union types instead of enums.
Model mutually exclusive states as discriminated unions with a stable discriminator; avoid boolean flags combined with optional payloads.
Construct discriminated-union transitions by explicitly listing target-branch fields; read unions withswitchand aneverexhaustiveness check.
Avoid unnecessaryascasts; narrow with type guards,inchecks, or records. Unavoidable casts require a// SAFETY:explanation.
Trace type mismatches to their source rather than casting at the consumer; verify whether the change introduced the mismatch.
Do not annotate or provide explicit type arguments when the compiler can infer them.
Validate large-union object literals withas const satisfies T, not a: Tannotation.
Use.at(0)when an element may be absent; use[0]only when existence is established or documented with// SAFETY:.
Skip barrel files such asindex.ts; import from explicit module paths.
Prefer arrow functions over function expressions.
Destructure function parameters when the intermediate parameter variable is not reused.
Prefer discriminated-union checks such asobj.type === "x"; useinonly when no ...
Files:
apps/api/src/lib/compression.test.tsapps/api/src/lib/corpus-index/chunking.test.tsapps/api/src/lib/limits.tsapps/api/src/lib/compression.tsapps/api/src/handlers/case-law/corpus-storage.tsapps/api/src/lib/corpus-index/chunking.ts
apps/**
📄 CodeRabbit inference engine (CONTRIBUTING.md)
apps/is reserved for runnable applications only.
Files:
apps/api/src/lib/compression.test.tsapps/api/src/lib/corpus-index/chunking.test.tsapps/api/src/lib/limits.tsapps/api/src/lib/compression.tsapps/api/src/handlers/case-law/corpus-storage.tsapps/api/src/lib/corpus-index/chunking.ts
!**/node_modules/**
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Do not open a public issue for security vulnerabilities; report them privately to
security@stll.app.
Files:
apps/api/src/lib/compression.test.tsapps/api/src/lib/corpus-index/chunking.test.tsapps/api/src/lib/limits.tsapps/api/src/lib/compression.tsapps/api/src/handlers/case-law/corpus-storage.tsapps/api/src/lib/corpus-index/chunking.ts
**/*
📄 CodeRabbit inference engine (AGENTS.md)
**/*: Treat legal data, personal data, and repository secrets as sensitive; do not expose private or internal context in generated repository artifacts.
Preserve least privilege, auditability, data minimization, workspace isolation, explicit access checks, and encryption-aware handling.
Use concise public engineering context in PRs, commits, comments, and documentation; do not include private business or security architecture details.
When commenting on GitHub, appendCC on behalf of usernameusing plain text without@.
Use scoped workspace filters such asbun --filter@stll/web...when running package commands.
Files:
apps/api/src/lib/compression.test.tsapps/api/src/lib/corpus-index/chunking.test.tsapps/api/src/lib/limits.tsapps/api/src/lib/compression.tsapps/api/src/handlers/case-law/corpus-storage.tsapps/api/src/lib/corpus-index/chunking.ts
**/*.{ts,tsx,js,jsx,md,mdx}
📄 CodeRabbit inference engine (AGENTS.md)
Keep prose concise and prefer varied punctuation over em dashes.
Files:
apps/api/src/lib/compression.test.tsapps/api/src/lib/corpus-index/chunking.test.tsapps/api/src/lib/limits.tsapps/api/src/lib/compression.tsapps/api/src/handlers/case-law/corpus-storage.tsapps/api/src/lib/corpus-index/chunking.ts
apps/api/src/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/api/src/**/*.ts: Never return unbounded result sets; filter queries by tenant ID and keep APIs stateless.
List endpoints must use cursor pagination and the standardPage<T>envelope{ items, nextCursor, limit }; offset pagination,totalCount, and unbounded lists require explicit justification.
Files:
apps/api/src/lib/compression.test.tsapps/api/src/lib/corpus-index/chunking.test.tsapps/api/src/lib/limits.tsapps/api/src/lib/compression.tsapps/api/src/handlers/case-law/corpus-storage.tsapps/api/src/lib/corpus-index/chunking.ts
**/*.{test,spec}.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Test behavior the type system, framework, or linter could miss; prefer invariants over examples for large input spaces.
Files:
apps/api/src/lib/compression.test.tsapps/api/src/lib/corpus-index/chunking.test.ts
apps/api/**/*.ts
📄 CodeRabbit inference engine (apps/api/AGENTS.md)
apps/api/**/*.ts: Do not export types that have no consumer; Elysia and Eden infer handler types, so manually exported schema types are unnecessary unless imported elsewhere.
Prefer Bun-native APIs such asBun.CryptoHasher,Bun.file, andBun.S3Clientover Web Crypto or manual implementations; do not write browser-compatible backend code.
Use Drizzle ORM for all database access.
Do not use?.or?? []to silently handle relations that are structural data-model invariants; usepanic()instead.
Apply timeouts to all external calls, such asfetch(url, { signal: AbortSignal.timeout(10_000) }).
Validate inputs at the boundary with Valibot or Elysia schemas.
Use Elysiatfor HTTP route contracts, Valibot for web and general runtime validation, and Zod only when explicitly required by a dependency.
At untrusted boundaries, preferv.strictObject()overv.object()unless stripping unknown keys is intentional.
Derive related Valibot schemas withv.pick(),v.omit(), andv.partial()instead of manually rewriting sibling schemas.
For cross-field form rules, preferv.partialCheck()withv.forward()so errors attach to the relevant field.
Put normalization inside Valibot schemas; usev.InferInputfor raw form values andv.InferOutputafter parsing and normalization.
Prefer declarative or built-in validators such ast.File({ maxSize }),v.isoDate(), andv.email()over manual checks.
Do not add fallback values for properties guaranteed by the framework or internal code; add defensive fallbacks only at true system boundaries.
Endpoint modules should default-export one{ config, handler }object. Handler-level concerns belong inconfig; reusable helpers must be in separate modules.
Create backend handlers withcreateSafeHandlerfor workspace scope orcreateSafeRootHandlerfor root scope fromapps/api/src/lib/api-handlers.ts; useResult.gen(),async function*,yield* Result.await(safeDb(...)), and `Result.err(new HandlerError(......
Files:
apps/api/src/lib/compression.test.tsapps/api/src/lib/corpus-index/chunking.test.tsapps/api/src/lib/limits.tsapps/api/src/lib/compression.tsapps/api/src/handlers/case-law/corpus-storage.tsapps/api/src/lib/corpus-index/chunking.ts
🧠 Learnings (7)
📚 Learning: 2026-06-30T14:05:30.529Z
Learnt from: jan-kubica
Repo: stella/stella PR: 927
File: packages/folio/src/core/ai-edits/aiEdits.test.ts:1379-1379
Timestamp: 2026-06-30T14:05:30.529Z
Learning: When reviewing TypeScript/JavaScript, do not treat reuse of the same local variable name (e.g., `const runs`) as a duplicate declaration if each instance is declared in a different lexical/function scope—such as separate `test()` callback bodies. Only flag duplicate declarations when they occur within the same lexical scope/block (where redeclaration is actually illegal or confusing), e.g., two `const` declarations for the same identifier in the same `{ ... }` block or same function scope.
Applied to files:
apps/api/src/lib/compression.test.tsapps/api/src/lib/corpus-index/chunking.test.tsapps/api/src/lib/limits.tsapps/api/src/lib/compression.tsapps/api/src/handlers/case-law/corpus-storage.tsapps/api/src/lib/corpus-index/chunking.ts
📚 Learning: 2026-07-20T08:56:12.738Z
Learnt from: jan-kubica
Repo: stella/stella PR: 746
File: .agents/plans/046-ai-memory.md:0-0
Timestamp: 2026-07-20T08:56:12.738Z
Learning: Firm-wide AI memory write operations (creating/updating firm memory) must be authorized using the dedicated `firmMemory` permission with `create` and `update` actions. In the authorization logic, `firmMemory` should be mapped to the `owner` and `admin` roles. Do not treat `organizationSettings.update` as an authorization fallback for any firm-memory `create`/`update` checks; authorization for firm-memory writes should fail unless `firmMemory` is present with the required action.
Applied to files:
apps/api/src/lib/compression.test.tsapps/api/src/lib/corpus-index/chunking.test.tsapps/api/src/lib/limits.tsapps/api/src/lib/compression.tsapps/api/src/handlers/case-law/corpus-storage.tsapps/api/src/lib/corpus-index/chunking.ts
📚 Learning: 2026-07-10T10:03:48.996Z
Learnt from: jan-kubica
Repo: stella/stella PR: 1076
File: .oxlint-plugins/__fixtures__/require-use-shallow.fixture.tsx:95-103
Timestamp: 2026-07-10T10:03:48.996Z
Learning: In the stella/stella repo, `unicorn/prefer-at` is configured without `checkAllIndexAccess`, so it only flags from-the-end indexing patterns (e.g., `arr[arr.length - 1]`) and not literal zero-index reads like `arr[0]`. During reviews, do not suggest converting `arr[0]` to `arr.at(0)` solely based on this rule; only recommend `at()` conversions when the existing rule would actually report an issue.
Applied to files:
apps/api/src/lib/compression.test.tsapps/api/src/lib/corpus-index/chunking.test.tsapps/api/src/lib/limits.tsapps/api/src/lib/compression.tsapps/api/src/handlers/case-law/corpus-storage.tsapps/api/src/lib/corpus-index/chunking.ts
📚 Learning: 2026-07-14T22:02:31.658Z
Learnt from: jan-kubica
Repo: stella/stella PR: 1175
File: apps/api/src/handlers/style-sets/replace.ts:38-41
Timestamp: 2026-07-14T22:02:31.658Z
Learning: In the Bun/Elysia backend under apps/api (Drizzle ORM using Relational Query Builder v2 via defineRelations), object-form `where` filters in RQBv2 queries are valid and should not be flagged. Specifically, for `db.query.<table>.findFirst()` / `findMany()`, use (and reviewers should accept) `where` shaped like `{ id: { eq: value }, organizationId: { eq: value } }`. Do not confuse this with the older v1 callback-style `where` API (e.g., `where: (table, { eq }) => ...`).
Applied to files:
apps/api/src/lib/compression.test.tsapps/api/src/lib/corpus-index/chunking.test.tsapps/api/src/lib/limits.tsapps/api/src/lib/compression.tsapps/api/src/handlers/case-law/corpus-storage.tsapps/api/src/lib/corpus-index/chunking.ts
📚 Learning: 2026-07-14T22:02:32.634Z
Learnt from: jan-kubica
Repo: stella/stella PR: 1175
File: apps/api/src/lib/style-sets.ts:94-97
Timestamp: 2026-07-14T22:02:32.634Z
Learning: In this codebase’s Drizzle ORM usage (Relational Queries v2 via `defineRelations`, typically `db.query.<table>.findFirst(...)`), allow object-form `where` filters like `where: { id: { eq: value }, organizationId: { eq: value } }`. Do not flag this as an invalid Drizzle/Prisma-incompatible pattern—this object+`eq` syntax is supported by Drizzle’s v2 relational query API and should be treated as correct.
Applied to files:
apps/api/src/lib/compression.test.tsapps/api/src/lib/corpus-index/chunking.test.tsapps/api/src/lib/limits.tsapps/api/src/lib/compression.tsapps/api/src/handlers/case-law/corpus-storage.tsapps/api/src/lib/corpus-index/chunking.ts
📚 Learning: 2026-07-29T10:59:41.806Z
Learnt from: jan-kubica
Repo: stella/stella PR: 1450
File: apps/web/src/routes/_protected.knowledge/-components/import-skill-dialog.tsx:95-110
Timestamp: 2026-07-29T10:59:41.806Z
Learning: When reviewing Elysia route handlers that use the `invalidateQuery: true` option, treat `queryKey` as a legitimate required handler body field. The `invalidateQuery` macro composes the macro’s body schema (from `apps/api/src/lib/invalidate-query-macro.ts`) with the handler’s local body schema, and the macro’s `afterHandle` reads `queryKey` to broadcast cache invalidation. Therefore, do not flag `queryKey` as an unsupported handler-body field; instead, verify that callers provide the required `queryKey` value.
Applied to files:
apps/api/src/lib/compression.test.tsapps/api/src/lib/corpus-index/chunking.test.tsapps/api/src/lib/limits.tsapps/api/src/lib/compression.tsapps/api/src/handlers/case-law/corpus-storage.tsapps/api/src/lib/corpus-index/chunking.ts
📚 Learning: 2026-07-18T08:39:56.674Z
Learnt from: jan-kubica
Repo: stella/stella PR: 1227
File: apps/api/src/handlers/entities/delete-version.test.ts:0-0
Timestamp: 2026-07-18T08:39:56.674Z
Learning: In the TypeScript API, a `ScopedDb` created via `createMembershipScopedDb` runs in its own short-lived RLS transaction per invocation. Do not use a “validation” query in one `scopedDb` call to authorize a read performed in a later `scopedDb` call—concurrent state changes (e.g., an entity-version tombstone) can make the later read invalid. Instead, combine the dependent reads into a single `scopedDb` invocation/transaction, or enforce the required state predicate in the content query itself (so RLS filtering and correctness are applied in the same transaction/read).
Applied to files:
apps/api/src/handlers/case-law/corpus-storage.ts
🔇 Additional comments (6)
apps/api/src/lib/compression.ts (1)
1-32: LGTM!apps/api/src/lib/compression.test.ts (1)
1-25: LGTM!apps/api/src/lib/limits.ts (1)
272-276: LGTM!apps/api/src/handlers/case-law/corpus-storage.ts (1)
9-12: LGTM!Also applies to: 30-31, 164-184, 210-210, 222-222, 236-236
apps/api/src/lib/corpus-index/chunking.ts (1)
1-1: LGTM!Also applies to: 92-106, 234-236, 319-323, 362-368
apps/api/src/lib/corpus-index/chunking.test.ts (1)
450-510: LGTM!
Review round: the decompressed-size ceiling now uses node:zlib's maxOutputLength so an oversized frame is rejected while decoding instead of after materializing, and the chunker budgets total AST characters so one giant block cannot bypass the block-count ceiling.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fe5732cdc4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "corpus-write-text", | ||
| async () => | ||
| await s3.write(keys.textKey, zstdCompress(text ?? ""), { | ||
| await s3.write(keys.textKey, await zstdCompressAsync(text ?? ""), { |
There was a problem hiding this comment.
In canonical or dual-write ingestion, if the Bun worker pool is backed up long enough for zstdCompressAsync to exceed the 60-second corpus timeout, boundedCorpusIo rejects before s3.write has started; the abandoned closure later resumes after withTimeout has cleared its timer and issues a PUT with no repository-enforced timeout. This can retain a stuck S3 request indefinitely, and on the new-row canonical path cleanup can finish before that late PUT and leave an orphan, so compress before entering boundedCorpusIo and give the PUT its own full timeout.
AGENTS.md reference: apps/api/AGENTS.md:L17-L21
Useful? React with 👍 / 👎.
1 participant
The corpus indexer isolates per-row failures into failed index jobs, but that isolation only catches code that throws. A document whose synchronous cost is unbounded (a corrupt many-block AST, a giant decompressed payload, malformed heading levels making path materialization quadratic) never throws — it blocks the daemon's event loop until the starvation watchdog restarts the process, and the same row is re-picked on the next start.
This bounds every unbounded step so such a document becomes a catchable, audited failure:
corpusPayloadMaxDecompressedBytesbefore the string reachesJSON.parseor the chunker. Existing sync-written objects decompress byte-for-byte through the bounded path (covered by test).everyconformance scan), plus a heading-ancestry depth clamp.Follow-up (not here): failed rows are retried each cycle; the retries are now cheap and visible, but an attempt-capped quarantine would stop them occupying batch slots.
Summary by CodeRabbit
New Features
Bug Fixes