fix(ai): validate generated MDX syntax#158
Merged
Merged
Conversation
📝 WalkthroughWalkthroughThis PR adds Studio MDX parser-based validation to the AI proposal validator. Proposals with body MDX content are now parsed using the same parser Studio uses, rejecting proposals with syntax errors. Validation applies to all operation types (create_document, delete_document, replace_selection, insert_block). Three MDX parsing libraries are added as dependencies. ChangesMDX Parsing Validation in Proposals
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint skipped: no ESLint configuration detected in root package.json. To enable, add Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
packages/modules/core.ai/src/server/validate-proposal.test.ts (1)
433-467: ⚡ Quick winBroaden MDX parse-failure regression coverage across MDX-writing operations.
This new test is solid for
insert_block; please add matching parse-failure cases forcreate_document.bodyandreplace_selection.replacementTextso the new cross-operation contract is locked in.Suggested test pattern
+ test("rejects create_document body MDX that the Studio parser cannot parse", async () => { + const validator = createSchemaAwareProposalValidator({ schemaLookup: lookup }); + const result = await validator( + createCandidate({ + type: "page", + operations: [ + { + op: "create_document", + path: "pages/feature-grid", + format: "mdx", + frontmatter: { title: "Feature Grid" }, + body: '<FeatureGrid features=\'[{"title":"Drop","description":"Once they\\\'re gone, they\\\'re gone."}]\' />', + }, + ], + }), + ); + assert.equal(result.status, "invalid"); + if (result.status === "invalid") { + assert.ok(result.errors.some((e) => e.code === "MDX_PARSE_FAILED" && e.path === "operations[0].body")); + } + }); + + test("rejects replace_selection replacementText MDX that the Studio parser cannot parse", async () => { + const validator = createSchemaAwareProposalValidator({ schemaLookup: lookup }); + const result = await validator({ + proposalId: "p1", + kind: "replace_selection", + project: "demo", + environment: "draft", + type: "page", + locale: "en", + summary: "Replace block", + operations: [ + { + op: "replace_selection", + selectionId: "sel_1", + originalText: "old", + replacementText: + '<FeatureGrid features=\'[{"title":"Drop","description":"Once they\\\'re gone, they\\\'re gone."}]\' />', + }, + ], + expiresAt: "2026-05-15T00:05:00.000Z", + provider: { providerId: "echo", model: "echo-1", promptTemplateId: "chat_tools.v1" }, + }); + assert.equal(result.status, "invalid"); + if (result.status === "invalid") { + assert.ok(result.errors.some((e) => e.code === "MDX_PARSE_FAILED" && e.path === "operations[0].replacementText")); + } + });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/modules/core.ai/src/server/validate-proposal.test.ts` around lines 433 - 467, Add two more tests mirroring the existing "rejects insert_block MDX..." case: create one that builds a proposal with kind "create_document" (or operation.op "create_document") using an invalid MDX string in create_document.body and another with kind "replace_selection" (operation.op "replace_selection") using an invalid MDX string in replace_selection.replacementText; call the same createSchemaAwareProposalValidator/validator and assert result.status === "invalid", then assert the first error has code "MDX_PARSE_FAILED", path "create_document.body" (for the create_document case) or "operations[0].replacementText" (for replace_selection) and that the error.message matches /Studio MDX parser/ to match the existing pattern.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@packages/modules/core.ai/src/server/validate-proposal.test.ts`:
- Around line 433-467: Add two more tests mirroring the existing "rejects
insert_block MDX..." case: create one that builds a proposal with kind
"create_document" (or operation.op "create_document") using an invalid MDX
string in create_document.body and another with kind "replace_selection"
(operation.op "replace_selection") using an invalid MDX string in
replace_selection.replacementText; call the same
createSchemaAwareProposalValidator/validator and assert result.status ===
"invalid", then assert the first error has code "MDX_PARSE_FAILED", path
"create_document.body" (for the create_document case) or
"operations[0].replacementText" (for replace_selection) and that the
error.message matches /Studio MDX parser/ to match the existing pattern.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 91a076f6-ac06-444a-8c58-8aac5041e59c
⛔ Files ignored due to path filters (1)
bun.lockis excluded by!**/*.lock
📒 Files selected for processing (4)
.ai/memory/lessons.mdpackages/modules/core.ai/src/server/validate-proposal.test.tspackages/modules/core.ai/src/server/validate-proposal.tspackages/modules/package.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test Plan
Summary by CodeRabbit
Release Notes
Bug Fixes
Tests