Skip to content

About

Full-stack AI research workflow system using Next.js, NestJS, and MongoDB with traceable multi-step reasoning

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Marginalia

Ask questions of your own documents. A small team of specialists — a planner, a retriever, a summarizer, a fact-checker and a writer — researches your library and returns an answer in which every claim is cited, with contradictions between sources flagged.

The research desk

A finished inquiry Dark mode
Inquiry with citations Inquiry in dark mode

UI previews. The answer text in these screenshots comes from a scripted demo run.

Features

  • Streaming research. Watch the five stages run live; the answer streams in word by word (Server-Sent Events). Stop a run at any time.
  • Grounded, cited answers. The writer may only use retrieved passages and cites them as [n]. Hover a citation to highlight its source in the margin; uncited sources are dimmed.
  • Fact-checking. A structured-output cross-check flags claims where your sources disagree.
  • A real library. Upload PDF, Markdown or text files (drag and drop, several at once) or paste text. Documents are split into passages and indexed with MongoDB full-text search. Search, filter by topic, edit (re-indexes automatically) and delete.
  • Inquiry history. Every run — including failed and stopped ones — is saved with its full trace and has a permalink.
  • Accounts. Sign-in and sign-up with Clerk. Every document, passage and inquiry belongs to one user; nobody can see, search or cite anyone else's library. New users can load the sample documents in one click.
  • Production basics. Per-user auth, a service API key, per-user rate limiting, Helmet security headers, request validation, structured errors, /health with real dependency checks, Swagger docs, request logging, graceful shutdown, Docker images and CI.
  • Thoughtful UI. Editorial design with light/dark themes, keyboard shortcuts (N new inquiry, Ctrl/⌘ + Enter to submit), responsive down to phone width, accessible focus states and live regions.

How it works

 question ─► 1 Plan ──► 2 Retrieve ──► 3 Summarize ──┬─► 4 Cross-check ─┐
             split into  $text search    extractive   │   Gemini, JSON   ├─► saved trace
             sub-questions over passages notes        └─► 5 Write ───────┘   + permalink
                                                          Gemini, streamed, cited [n]
  1. Plan — rule-based splitting of common shapes ("Compare X vs Y", "Should I use X or Y…", "Pros and cons of X"…) into focused sub-questions.
  2. Retrieve — each sub-question searches the chunks text index (stemmed, title-weighted). Results are de-duplicated across sub-questions, limited to two passages per document for diversity, and capped at eight.
  3. Summarize — an extractive summary of each passage for the sources margin.
  4. Cross-check and 5. Write run in parallel: Gemini compares the passages for contradictions (constrained to a JSON schema) while a second call streams the cited answer. A failed cross-check does not fail the run.

If nothing in the library matches, the team says so instead of letting the model answer from memory.

Architecture

Browser ──► Next.js (client/) ── /api/* proxy ─────────────────►  NestJS API (server/)  ──► MongoDB
   ▲        Clerk session,       adds Authorization: Bearer        verifies token · owner   docs, chunks,
   │        UI, RTK Query        <session JWT> + x-api-key         scoping · rate limit ─►  traces · Gemini
   └──────► Clerk (sign-in, sign-up, account)

The browser only ever talks to the Next.js app. Its /api/* route handler streams requests (including file uploads and SSE) to the NestJS API, adding the signed-in user's Clerk session token and the server-side API_KEY. Neither the key nor the backend address reaches the browser, and no CORS configuration is needed.

Authentication and data ownership

  • Pages call auth.protect(), so signed-out visitors are redirected to /sign-in and returned afterwards.
  • The API verifies every request itself (it doesn't trust the proxy): UserAuthGuard checks the Clerk session JWT with @clerk/backend — signature, expiry and, if CLERK_AUTHORIZED_PARTIES is set, the issuing origin. Only / and /health are public.
  • Every query is scoped by owner. Documents, passages and inquiries carry an ownerId; another user's ids return 404, never 403, so they can't be probed. Retrieval only ever searches the caller's own passages.
  • Rate limits are per user, not per IP, so people behind one proxy don't share a budget.
  • API_KEY remains as a service-to-service lock so only your Next.js server can reach the API.
  • Upgrading an existing install: data created before accounts existed is assigned to the first user who signs in, and the old indexes are replaced automatically at startup.

Tech stack

Layer Technology
Frontend Next.js 16 (App Router, Turbopack), React 19, Tailwind CSS 4, Redux Toolkit Query, Sonner
Auth Clerk (@clerk/nextjs in the UI, @clerk/backend token verification in the API)
Backend NestJS 12, Express 5, Mongoose 9, class-validator, @nestjs/throttler, Helmet, Swagger
AI Google Gemini via @google/genai (streaming + structured JSON output)
Files pdf-parse for PDFs; Markdown and plain text natively
Database MongoDB 8 (text indexes for retrieval and library search)
Tooling TypeScript 6, ESLint 10, Prettier, Jest 30 + Supertest, Vitest + Testing Library, Docker

Repository layout

.
├── client/                    Next.js app
│   ├── app/                   routes: / (desk), /research/[id], /library, /library/[id], /upload
│   │   └── api/[...path]/     same-origin proxy to the API
│   ├── components/            research/, library/, upload/, shell/, ui.tsx
│   ├── lib/                   API client, streaming run state, SSE parser, citations, theming
│   └── Dockerfile
├── server/                    NestJS API
│   ├── src/
│   │   ├── workflow/          planner, retriever, summarizer, cross-checker, Gemini, prompts, SSE controller
│   │   ├── docs/              documents + chunks, file parsing, upload, chunking
│   │   ├── health/            /health
│   │   ├── common/            API-key guard, request logger
│   │   ├── config/            environment validation
│   │   └── seed/              seed script
│   ├── seed_docs/             sample Markdown documents (front matter: title, topic, source)
│   ├── test/                  e2e tests (real MongoDB, fake Gemini)
│   └── Dockerfile
├── docker-compose.yml         MongoDB for development; full stack with --profile app
└── .github/workflows/ci.yml   lint, typecheck, unit, e2e, build, Docker images

Quick start

Prerequisites: Node.js 22.12+ (24 recommended, see .nvmrc), MongoDB (local, Atlas, or docker compose up -d mongodb), a Gemini API key from https://aistudio.google.com/apikey, and a free Clerk application from https://dashboard.clerk.com (Configure → API keys).

npm run setup                               # install root, server and client dependencies

cp server/.env.example server/.env          # set GEMINI_API_KEY and CLERK_SECRET_KEY
cp client/.env.example client/.env.local    # set NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY and CLERK_SECRET_KEY

npm run dev                                 # API on :4000, UI on :3000

Open http://localhost:3000, create an account, and use Load sample documents on the empty library to get 11 documents to research. API docs are at http://localhost:4000/api-docs.

Use the same Clerk application's keys in both files. In a Clerk development instance you can sign up with anything+clerk_test@example.com and the verification code 424242.

On Windows use copy instead of cp.

Running with Docker

cp .env.example .env                        # set API_KEY, the Clerk keys and GEMINI_API_KEY
npm run docker:up                           # = docker compose --profile app up -d --build

The UI is served on http://localhost:3000. The API is published on 127.0.0.1:4000 only, and MongoDB on 127.0.0.1:27017. Both images are multi-stage, run as a non-root user and define health checks.

Configuration

Server (server/.env)

Variable Default Description
NODE_ENV development production makes API_KEY mandatory
PORT 4000 Listen port
MONGODB_URI mongodb://localhost:27017/mini_research MongoDB connection string
GEMINI_API_KEY (empty) Required for research; the library works without it
GEMINI_MODEL gemini-3.8-flash Model id, or gemini-flash-latest
CLERK_SECRET_KEY (required) Verifies users' session tokens (or set CLERK_JWT_KEY, the PEM public key, for networkless checks)
CLERK_AUTHORIZED_PARTIES (any) Comma-separated UI origins allowed to have issued sessions, e.g. http://localhost:3000
API_KEY (empty = open) Shared secret expected in the x-api-key header
TRUST_PROXY false Express trust proxy (true, hop count, loopback, uniquelocal…) so rate limits use client IPs
RATE_LIMIT_PER_MINUTE 120 Requests per minute per IP, all routes
ASK_RATE_LIMIT_PER_MINUTE 10 Research runs per minute per IP
MAX_UPLOAD_MB 10 Largest accepted upload
SWAGGER_ENABLED true Serve interactive docs at /api-docs
CORS_ORIGIN (all) Only for browsers calling the API directly (not needed with the proxy)

Invalid values stop the server at startup with a readable message. Variables set in your shell or system environment take precedence over .env.

Client (client/.env.local)

Variable Default Description
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY (required) Clerk publishable key (safe to expose)
CLERK_SECRET_KEY (required) Clerk secret key, same application as the server
API_URL http://localhost:4000 Where the Next.js server reaches the API
API_KEY (empty) Must match the server's API_KEY; never sent to the browser

The older NEXT_PUBLIC_SERVER is still read as a fallback for API_URL.

Scripts

Run from the repository root:

Script Description
npm run setup Install root, server and client dependencies
npm run dev API and UI together, with live reload
npm run build Production builds of both apps
npm run start Start both production builds
npm run lint ESLint, both apps
npm run typecheck TypeScript, both apps
npm test Unit tests: Jest (server) and Vitest (client)
npm run test:e2e API end-to-end tests (needs MongoDB; Gemini is faked)
npm run seed -- <id> Load the sample docs into one user's library (Clerk id)
npm run format Prettier (format:check to verify)
npm run docker:up Build and start the full stack in Docker

API

All routes except / and /health require Authorization: Bearer <Clerk session token>, plus x-api-key when API_KEY is set. Everything is scoped to the token's user. Interactive documentation: /api-docs.

Method Path Description
GET /health ok / degraded (AI unavailable) / down (503, database down)
POST /ask Run the workflow for { question } and return { id, trace }
POST /ask/stream Same, streamed as Server-Sent Events (below)
GET /traces Recent inquiries (?limit=)
GET /traces/:id One inquiry with its full trace (/trace/:id is a legacy alias)
DELETE /traces/:id Delete an inquiry
POST /upload Add a document from JSON { title, topic, content, source? }
POST /upload/file Add a document from a multipart file (+ optional title, topic)
GET /docs Paginated list with ?q= full-text search, ?topic=, ?page=
GET /docs/topics Topics with document counts
GET /docs/stats Totals: documents, passages, topics, words
GET /docs/:id One document, including its content
POST /docs Create a document
POST /docs/samples Add the bundled sample documents to your library (idempotent)
PATCH /docs/:id Update a document (re-indexes its passages)
DELETE /docs/:id Delete a document and its passages

/ask/stream events, in order: start { id } → step (each stage as it starts and finishes) → sources → crosscheck and token (interleaved) → done (the saved trace) or error { statusCode, message }.

Errors use NestJS' shape: { statusCode, error, message }, where message is an array for validation errors.

Testing

  • Server unit tests (server/src/**/*.spec.ts): chunking, planner, Gemini error handling, cross-check sanitizing, the whole workflow with fakes, auth guard, env validation, file parsing (including a real PDF).
  • Server e2e tests (server/test/): the real app against MongoDB with Gemini and Clerk token checks faked — two users' data isolation (listing, fetching, editing, deleting, retrieval, history), the legacy-data claim, sample loading, API key and session auth, validation, uploads (JSON, Markdown, PDF), search and pagination, re-indexing, both ask endpoints including SSE, failure handling and cascading deletes. They use a separate mini_research_e2e database.
  • Client tests (*.test.ts(x)): SSE parsing, citations, formatting, the streaming run reducer, and the answer, composer and pipeline components.

Troubleshooting

  • The status in the sidebar says "AI unavailable" — click it for the reason. /health makes a real generation call, so it catches keys that can list models but can't generate.
  • PERMISSION_DENIED — the key's Google Cloud project was denied access. Create a new key.
  • 404 NOT_FOUND … no longer available — the configured GEMINI_MODEL was retired. Pick a current one.
  • Changes to server/.env seem ignored — a variable with the same name is set in your shell or system environment and wins. Check with echo $GEMINI_API_KEY (bash) or $env:GEMINI_API_KEY (PowerShell).
  • "Missing or invalid API key" — API_KEY in client/.env.local must match server/.env.
  • The server won't start: "CLERK_SECRET_KEY … is required" — add the key from the Clerk dashboard.
  • Every request returns "Your session has expired" — the server and client are using keys from different Clerk applications, or CLERK_AUTHORIZED_PARTIES doesn't include the URL you're browsing.
  • The UI says the research API is unreachable — the backend isn't running, or API_URL is wrong.

License

The packages are marked UNLICENSED. Add a license file if you plan to open-source the project.

About

Full-stack AI research workflow system using Next.js, NestJS, and MongoDB with traceable multi-step reasoning

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages