feat(storage-foundation): zod schemas, public + private stores, markdown pipeline#13
Merged
Conversation
…line + s3 sdk - apps/api: move gitsheets from devDependencies to dependencies (production store code imports it); add @aws-sdk/client-s3, @cfp/shared - packages/shared: add zod, zod-to-json-schema, unified, remark-parse, remark-gfm, remark-breaks, remark-rehype, rehype-sanitize, rehype-stringify, remark-stringify, strip-markdown Commands: npm install --workspace=apps/api gitsheets@1.0.3 npm install --workspace=apps/api @aws-sdk/client-s3 @cfp/shared npm install --workspace=packages/shared zod zod-to-json-schema unified remark-parse remark-gfm remark-breaks remark-rehype rehype-sanitize rehype-stringify remark-stringify strip-markdown
All 13 entities from specs/data-model.md — public (Person, Project, ProjectMembership, ProjectUpdate, ProjectBuzz, Tag, TagAssignment, HelpWantedRole, HelpWantedInterestExpression, SlugHistory, Revocation) and private (PrivateProfile, LegacyPasswordCredential). Each schema exports both the ZodObject and the inferred TypeScript type. Validators match field-level rules from specs/data-model.md (slug regex, max lengths, enum values, cross-field refinements for featured projects).
packages/shared/scripts/generate-json-schemas.ts generates JSON Schemas from Zod schemas using Zod v4's built-in toJSONSchema (output mode). $schema field is stripped since gitsheets uses ajv@8 in draft-07 mode. .gitsheets/schemas/<Entity>.schema.json — committed generated outputs. Check-schemas script verifies drift via `git diff --exit-code`. Command to regenerate: npm run generate-schemas -w packages/shared
packages/shared/src/markdown.ts implements renderMarkdown(source) returning
{ html, excerpt } per specs/behaviors/markdown-rendering.md.
Pipeline: remark-parse → remark-gfm → remark-breaks → remark-rehype
→ heading demotion (h1→h3, min h6) → img attrs (loading=lazy)
→ rehype-sanitize → rehype-stringify
Sanitization: strips <script>, javascript:, on* attrs, non-https image srcs,
style attrs, raw HTML. Allows http/https/mailto hrefs, https image src.
Excerpt: first-paragraph text via strip-markdown, capped at 280 chars
with word-boundary truncation (first paragraph approach — see Notes).
Public store (apps/api/src/store/public.ts): - openPublicStore(repoPath) returns typed gitsheets Store via openStore() - All 11 public sheets declared with typed validators (Zod → StandardSchemaV1) - Type cast used for Zod v4 compatibility (structural mismatch in FailureResult) Private store (apps/api/src/store/private/): - PrivateStore interface per specs/behaviors/private-storage.md - BasePrivateStore: in-memory load, indices, transact with rollback on failure - FilesystemPrivateStore: atomic temp-file-then-rename writes - S3PrivateStore: @aws-sdk/client-s3 backed (production) - PrivateStoreError for structured error reporting Dual-store coordinator (apps/api/src/store/store.ts): - Store class wraps both stores with transact() for cross-store mutations - writeOrder: 'private-first' | 'public-first' per use-case - Cross-store failure uses reconciliation approach (not git-revert) Boot loader (apps/api/src/store/boot.ts): - bootStores(env) fails loudly if either store unreachable - Selects filesystem or S3 backend from STORAGE_BACKEND env var
Schema tests (packages/shared/tests/schemas.test.ts): - Valid and invalid fixture per entity (13 schemas total, 52 tests) - Tests for cross-field refinements (featured project) and regex validators Markdown tests (packages/shared/tests/markdown.test.ts): - Renders heading, link, GFM table, code blocks, blockquotes, images - Sanitization: rejects <script>, javascript:, on* event attrs, non-https images - Heading demotion: h1→h3, h2→h4, caps at h6 - loading=lazy on images, excerpt truncation at word boundary Store tests (apps/api/tests/store.test.ts): - Boots Store against createTestRepo() + upserts/queries a Project - Path template verified to contain slug - Dual-store: upserts Person (public) + PrivateProfile (private) in one transact - Cross-store rollback: handler throw → no public commit, no private PUT - Dual-write: private flush fails after public commit → error thrown, rollback Validation: createTestPrivateStore shim retained; tests use real FilesystemPrivateStore instead (decision documented in plan Notes).
themightychris
added a commit
that referenced
this pull request
May 16, 2026
Verified all 9 validation criteria. Notable decisions documented in Notes: - Cross-store rollback: chose reconciliation (b), not git-revert (a). In-memory is rolled back on flush failure; reconcile-private-store.ts handles repair. - Zod v4 built-in toJSONSchema used instead of zod-to-json-schema (library doesn't support Zod v4 yet — issue #14 filed for cleanup). - StandardSchemaV1 type cast needed for Zod v4 ↔ gitsheets compatibility. - createTestPrivateStore shim retained as lighter alternative; new store tests use real FilesystemPrivateStore directly. - Sheet.defineIndex and reconcile-private-store.ts deferred to write-api plan.
The previous implementation checked hasPrivateMutations() before the handler ran — staging arrays were always empty at that point so the 'private-first' branch was unreachable. Callers silently fell through to public-first semantics. Fix: run the handler inside public.transact's callback, then flush private either inside the callback (private-first) or after public commits (.then, public-first). The flush decision is made after the handler has staged its mutations. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ite tests - Add assertion that the public commit DID land when private flush fails (the load-bearing claim of the reconciliation strategy: public is committed, private is orphaned, manual recovery needed) - Remove unused dualStore variable from the same test block - Fix the cross-instance persistence test: create a fresh FilesystemPrivateStore from the same dir and load() it, instead of querying the original instance (which was testing nothing about persistence) - Update makePrivateStore() to return dir so callers can create fresh instances - Remove unused openRepo import and SHEET_CONFIGS const - Add test for writeOrder: 'private-first' path verifying that when private flush fails inside the public.transact callback, the public tree is also not committed (protecting both sides, unlike public-first behavior) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…chema Rename _stripped to \$schema and add void \$schema to make the intent explicit: the field is destructured specifically to omit it from the output object. The _ prefix does not satisfy the project's lint config. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
themightychris
force-pushed
the
feat/storage-foundation
branch
from
b444536 to
574ba9f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
specs/data-model.mdinpackages/shared/src/schemas/— public and private, with field-level validators matching the spec (slug regex, max lengths, enum values, cross-field refinements)toJSONSchema; committed to.gitsheets/schemas/;check-schemasscript fails CI on driftapps/api/src/store/public.ts) —openPublicStore()via gitsheetsopenStore()with typed validators for all 11 public sheetsapps/api/src/store/private/) —PrivateStoreinterface,FilesystemPrivateStore(dev, atomic temp-file-then-rename),S3PrivateStore(production,@aws-sdk/client-s3); both follow load-at-boot + in-memory + PUT-on-mutation pattern;transact()with rollback on failureStoreclass) —store.transact()withwriteOrder: 'private-first' | 'public-first'per the atomicity spec; cross-store failure uses reconciliation (not git-revert) per spec decisionpackages/shared/src/markdown.ts) —renderMarkdown()via unified/remark/rehype-sanitize; heading demotion,loading="lazy"on images, sanitization of<script>,javascript:,on*attrs, non-https image src; plain-text excerpt at 280 charsapps/api/src/store/boot.ts) —bootStores(env)fails loudly if either store unreachabledevDependenciestodependenciesinapps/api(production store code imports it)createTestPrivateStoreshim retained — downstream tests use realFilesystemPrivateStoreinstead; shim kept as a lighter alternative for the original harness testTest plan
packages/shared)generate-schemasscript produces correct JSON Schema outputs (committed);check-schemasexits 1 on driftStorebooted againstcreateTestRepo(), upserts Project, queries back, path template contains slugstore.transactinserts Person (public) + PrivateProfile (private) in one atomic call; both stores reflect the changerenderMarkdown('# Hello\n[link](https://x.org)')produces sanitized HTML + plain-text excerpt<script>,javascript:,on*=, non-https image src (RFC-style negative tests)createTestPrivateStoreshim evaluated — downstream tests migrated toFilesystemPrivateStore(shim retained as lighter alternative for legacy harness test)🤖 Generated with Claude Code