LitLens

Understand your literature. Not just collect it.

LitLens is an AI-powered research literature analysis platform that transforms a pile of PDFs into structured, queryable knowledge. Upload your research question and papers — get thematic clusters, alignment scoring, cited Q&A, and gap detection in seconds.

Features

Feature	Description
Passwordless Authentication	Magic-link email login powered by NextAuth v5. No passwords to remember — just enter your email and click the link.
Project Management	Create, rename, and delete research projects (up to 5 per account). Titles are auto-generated from your research question via Gemini. Switch between projects from the sidebar.
PDF Upload & Extraction	Drag-and-drop PDF upload (up to 20 files). Text extracted via pdfplumber with Tesseract OCR fallback for scanned pages. Metadata (title, authors, year) is detected automatically.
Semantic Chunking & Embedding	Extracted text is split into overlapping 5-sentence chunks and embedded into 3,072-dimension vectors using the Gemini embedding model. Stored in PostgreSQL via pgvector.
Orientation View	Scores how well your library aligns with your research question (0–100). Surfaces strong coverage areas, notable gaps, and an AI-generated library summary.
Thematic Clustering	Automatically groups chunks into thematic clusters via K-means (auto-k selection). Each cluster receives an LLM-generated label and one-sentence summary.
Q&A with Citations (RAG)	Ask free-form questions answered by semantic search over your documents. Responses include source passages with relevance scores (High / Medium / Low). Full Q&A history is persisted.
Gap Detection	Identifies 3–6 research topics underrepresented in your library. Each gap includes a priority level (High / Medium), description, and suggested search terms (click to copy). Gaps can be marked as addressed.
Document Management	View, edit, tag, and delete documents. Tag documents as Highly Useful, Reviewing, or Not Useful. Inspect individual text chunks with a sliding detail panel and copy-all.
LLM Rate-Limit Resilience	Model fallback chain (gemini-2.5-flash-lite → gemini-2.5-flash → gemma-3-12b-it), batched embedding with delays, and exponential backoff retries for free-tier usage.
Progressive Web App	Installable on mobile and desktop. Service worker caches static assets with a stale-while-revalidate strategy. API calls are always network-first.
Responsive Design	Mobile sidebar overlay, bottom tab bar, responsive document table, and sliding detail panel. Works on phones, tablets, and desktops.
Animated Splash Screen	SVG book-scanning animation with status word cycling. Shows once per session.

Architecture

┌─────────────────────────────────────────────────┐
│                 Next.js Frontend                │
│  (React 19 · Tailwind CSS 4 · Zustand)         │
└──────────────────┬──────────────────────────────┘
                   │ HTTP / Next.js API Routes
┌──────────────────▼──────────────────────────────┐
│            Next.js API Layer (Node.js)          │
│  /api/projects  ·  /api/projects/[id]/*         │
│  Prisma ORM  ·  Google Gemini SDK               │
└───────┬──────────────────────────┬──────────────┘
        │ SQL + pgvector            │ HTTP
┌───────▼──────────┐   ┌───────────▼──────────────┐
│  PostgreSQL 16   │   │   Python FastAPI Service  │
│  + pgvector ext  │   │   PDF extract · cluster   │
└──────────────────┘   └──────────────────────────┘

Request flow for a new project:

User submits a research question and PDFs via the wizard.
The Next.js API forwards PDFs to the Python service for text extraction (pdfplumber + Tesseract OCR).
Extracted text is chunked and each chunk is embedded with the Gemini embedding model (3,072-dimension vectors).
Embeddings are stored in PostgreSQL with the pgvector extension.
The Python clustering service runs K-means over the embeddings and returns cluster assignments.
Cluster labels and summaries are generated by the Gemini LLM.
Orientation, gap detection, and Q&A endpoints query the vector store at request time.

Tech Stack

Frontend

Package	Version	Purpose
Next.js	16.2.1	React framework (App Router)
React	19.2.4	UI library
TypeScript	5	Type safety
Tailwind CSS	4	Utility-first styling
Zustand	5.0.12	Client state management
Lucide React	1.7.0	Icon set
Motion	12.38.0	Animation library

Backend (Node.js)

Package	Version	Purpose
Prisma	7.5.0	ORM & migrations
`@prisma/adapter-pg`	7.5.0	PostgreSQL adapter
`@google/generative-ai`	0.24.1	Gemini embeddings & LLM
NextAuth	5.0.0-beta.30	Passwordless authentication
Nodemailer	7.0.13	Magic-link email transport
pg	8.20.0	PostgreSQL client

Python Microservice

Package	Version	Purpose
FastAPI	0.115.6	HTTP framework
Uvicorn	0.34.0	ASGI server
pdfplumber	0.11.4	PDF text extraction
pytesseract	0.3.13	OCR for scanned PDFs
pdf2image	1.17.0	PDF-to-image conversion
scikit-learn	1.6.1	K-means clustering
NLTK	3.9.1	Text tokenisation
NumPy	2.2.3	Numerical computing

Infrastructure

Tool	Purpose
Docker & Docker Compose	Container orchestration
PostgreSQL 16 + pgvector	Vector-capable relational database

Prerequisites

Node.js ≥ 18 and npm ≥ 9
Docker ≥ 24 and Docker Compose ≥ 2
A Google Gemini API key (get one here)

Quick Start

# 1. Clone the repository
git clone https://github.com/trtlbby/litlens.git
cd litlens

# 2. Install JavaScript dependencies
npm install

# 3. Copy the example environment file and fill in your values
cp .env.example .env.local
# Edit .env.local — see Environment Variables below

# 4. Start PostgreSQL and the Python microservice
docker compose up -d

# 5. Run database migrations
npx prisma migrate deploy

# 6. Start the development server
npm run dev

Open http://localhost:3000 in your browser.

Environment Variables

Create a .env.local file in the project root. The following variables are required:

Variable	Description	Example
`DATABASE_URL`	PostgreSQL connection string	`postgresql://litlens:litlens_password@localhost:5432/litlens`
`GEMINI_API_KEY`	Google Gemini API key	`AIza...`
`AUTH_SECRET`	Random string used to encrypt sessions	`openssl rand -hex 16`
`NEXTAUTH_URL`	Canonical URL of your deployment	`http://localhost:3000`
`EMAIL_SERVER`	SMTP connection string for magic-link emails	`smtp://user:pass@smtp.example.com:587`
`EMAIL_FROM`	Sender address for magic-link emails	`LitLens <noreply@litlens.app>`

The following variables are optional:

Variable	Description	Default
`PYTHON_SERVICE_URL`	URL of the FastAPI microservice	`http://localhost:8000`
`EMBEDDING_MODEL`	Gemini embedding model name	`gemini-embedding-001`
`LLM_MODEL`	Primary Gemini LLM model	`gemini-2.5-flash-lite`

Note: The default Docker Compose setup exposes PostgreSQL on localhost:5432 with the credentials shown above. Change them in docker-compose.yml and your .env.local if needed. For local development without a real mail server, magic-link URLs are logged to the terminal.

Development

Available Scripts

npm run dev      # Start the Next.js dev server (http://localhost:3000)
npm run build    # Create a production build
npm run start    # Start the production server
npm run lint     # Run ESLint

Docker Compose Services

Service	Container	Port	Description
`db`	`litlens-db`	`5432`	PostgreSQL 16 + pgvector
`python-service`	`litlens-python`	`8000`	FastAPI PDF & clustering service

# Start all services
docker compose up -d

# View logs
docker compose logs -f

# Stop all services
docker compose down

# Stop and remove volumes (resets the database)
docker compose down -v

Database Migrations

# Apply all pending migrations
npx prisma migrate deploy

# Create a new migration after editing schema.prisma
npx prisma migrate dev --name <migration-name>

# Regenerate the Prisma client (runs automatically after npm install)
npx prisma generate

# Open Prisma Studio (visual DB browser)
npx prisma studio

Python Service (local, without Docker)

cd python-service
pip install -r requirements.txt
uvicorn main:app --reload --port 8000

Tesseract must also be installed on the host for OCR support:

macOS: brew install tesseract

Ubuntu/Debian: sudo apt-get install tesseract-ocr

Project Structure

litlens/
├── app/                            # Next.js App Router
│   ├── api/                        # API route handlers
│   │   ├── auth/[...nextauth]/     # NextAuth magic-link endpoints
│   │   └── projects/
│   │       ├── route.ts            # POST (create) · GET (list)
│   │       └── [id]/
│   │           ├── route.ts        # GET · PATCH · DELETE project
│   │           ├── orient/         # POST trigger · GET cached orientation
│   │           ├── ask/            # POST question · GET history
│   │           ├── gaps/           # POST analyse · GET gaps
│   │           │   └── [gapId]/    # PATCH toggle dismissed
│   │           └── documents/      # POST upload · GET list
│   │               └── [docId]/    # GET · PATCH · DELETE document
│   ├── new/                        # New-project wizard (2-step)
│   ├── project/[id]/               # Per-project pages
│   │   ├── layout.tsx              # Sidebar + top bar + bottom tabs
│   │   ├── page.tsx                # Orientation dashboard
│   │   ├── ask/                    # Q&A interface
│   │   ├── documents/              # Document manager
│   │   └── gaps/                   # Gap detection
│   ├── layout.tsx                  # Root layout (fonts, PWA meta, AuthProvider)
│   ├── page.tsx                    # Landing page
│   └── globals.css                 # Design tokens + global styles
├── components/
│   ├── auth/                       # AuthContext, AuthGate, LoginModal, UserMenu
│   ├── documents/                  # DocumentPanel (sliding detail viewer)
│   ├── projects/                   # ProjectSwitcher dropdown
│   ├── ui/                         # Buttons, header, logos, splash, stepper, SWRegister
│   └── upload/                     # Dropzone, file-list, processing-screen
├── lib/
│   ├── auth.ts                     # NextAuth v5 config + project access helper
│   ├── openai.ts                   # Gemini SDK (embeddings + LLM + fallback chain)
│   ├── chunker.ts                  # Sliding-window text chunker
│   ├── prisma.ts                   # Prisma client singleton
│   └── stores/upload-store.ts      # Zustand upload state
├── generated/prisma/               # Auto-generated Prisma client code
├── prisma/
│   ├── schema.prisma               # Database schema (pgvector)
│   └── migrations/                 # SQL migration files
├── python-service/
│   ├── main.py                     # FastAPI app (/extract, /cluster, /health)
│   ├── extractor.py                # PDF text + metadata extraction
│   ├── clusterer.py                # K-means with auto-k selection
│   ├── requirements.txt            # Python dependencies
│   └── Dockerfile                  # Container definition
├── public/                         # PWA icons, manifest.json, sw.js
├── docker-compose.yml              # PostgreSQL + Python service
├── next.config.ts                  # Next.js configuration
├── tsconfig.json                   # TypeScript configuration
└── package.json                    # NPM manifest

API Reference

All API routes sit under /api. Authentication endpoints are handled automatically by NextAuth. Project endpoints are under /api/projects.

Authentication

Method	Path	Description
`GET/POST`	`/api/auth/*`	NextAuth magic-link sign-in, callback, sign-out (handled automatically)

Projects

`POST /api/projects`

Create a new project. Title is auto-generated from the research question via Gemini if not provided.

Request body

{
  "research_question": "string",
  "scope_context": "string (optional)",
  "methodology": "string (optional)",
  "known_coverage": "string (optional)"
}

Response — 201 Created

{ "id": "uuid", "title": "string", "research_question": "string", "created_at": "ISO 8601" }

`GET /api/projects`

List all projects for the authenticated user. Returns name, research question, file count, and timestamps.

`GET /api/projects/:id`

Fetch full project details including documents, clusters, and chunk count.

`PATCH /api/projects/:id`

Rename a project.

Request body

{ "title": "string" }

`DELETE /api/projects/:id`

Delete a project and all associated data (documents, chunks, clusters, gaps, Q&A history).

Documents

`POST /api/projects/:id/documents`

Upload a single PDF. Triggers extraction → chunking → embedding pipeline.

Request — multipart/form-data with a file field (PDF only).

Response — 201 Created

{ "id": "uuid", "project_id": "uuid", "filename": "paper.pdf", "title": "string", "authors": "string", "year": 2024, "chunk_count": 42, "created_at": "ISO 8601" }

Returns 207 Multi-Status if the document is saved but embedding fails (document is visible but not searchable until re-embedded).

`GET /api/projects/:id/documents`

List all documents in a project with metadata and chunk counts.

`GET /api/projects/:id/documents/:docId`

Fetch a single document with all its text chunks.

`PATCH /api/projects/:id/documents/:docId`

Update a document's title or relevance tag.

Request body

{ "title": "string (optional)", "tag": "string (optional)" }

`DELETE /api/projects/:id/documents/:docId`

Remove a document and its chunks from the project.

Orientation

`POST /api/projects/:id/orient`

Run the full orientation pipeline: cluster embeddings via K-means → label clusters with Gemini → compute alignment score → generate library summary. Long-running (up to 300 s).

Response — 200 OK

{
  "alignment_score": 82,
  "library_summary": "string",
  "strong_coverage": ["topic A", "topic B"],
  "notable_gaps": ["topic C"],
  "clusters": [
    { "cluster_index": 0, "label": "string", "summary": "string", "doc_count": 5, "doc_names": ["paper.pdf"] }
  ]
}

`GET /api/projects/:id/orient`

Fetch the cached orientation results (clusters, alignment score, coverage breakdown) without re-running the analysis.

Q&A

`POST /api/projects/:id/ask`

Ask a free-form question. Embeds the question, runs cosine-similarity search over chunk vectors (top 8), and generates a grounded answer via Gemini.

Request body

{ "question": "string (min 3 characters)" }

Response — 200 OK

{
  "id": "uuid",
  "question": "string",
  "answer": "string",
  "created_at": "ISO 8601",
  "sources": [
    { "id": "uuid", "passage": "string", "document_filename": "string", "document_title": "string", "relevance_score": 0.92, "relevance": "High" }
  ]
}

Relevance tiers: High (> 0.5), Medium (0.3–0.5), Low (< 0.3).

`GET /api/projects/:id/ask`

Fetch the full Q&A history for a project, including stored source passages.

Gap Detection

`POST /api/projects/:id/gaps`

Run gap analysis. Sends cluster summaries and the research question to Gemini, which identifies 3–6 underrepresented topics with priorities and suggested search terms.

Response — 200 OK

{
  "gaps": [
    { "id": "uuid", "title": "string", "priority": "HIGH", "description": "string", "searchTerms": ["term1", "term2"], "addressed": false }
  ]
}

`GET /api/projects/:id/gaps`

Fetch stored gaps and a coverage breakdown (percentage per cluster, visualised as a bar chart on the frontend).

`PATCH /api/projects/:id/gaps/:gapId`

Toggle a gap's dismissed/addressed status.

Database Schema

LitLens uses PostgreSQL with the pgvector extension for storing and querying 3,072-dimension embeddings.

Project ──< Document ──< Chunk >──< ChunkCluster >── Cluster
   │                       │
   ├──< Gap                └──< QaSource >── QaSession
   └──< QaSession

Table	Description
`projects`	Research project with question, scope, and LLM-generated summary.
`documents`	Uploaded PDFs with extracted metadata (title, authors, year, BibTeX).
`chunks`	Sentence-level text segments with 3,072-dim vector embeddings.
`clusters`	Thematic groups discovered via K-means with LLM-generated labels.
`chunk_clusters`	Many-to-many mapping between chunks and clusters with distance scores.
`gaps`	Missing topics identified by the LLM relative to the research question.
`qa_sessions`	Q&A exchanges (question + answer pairs).
`qa_sources`	Supporting passages with relevance scores, linked to sessions and chunks.
`users`	Authenticated users (NextAuth).
`accounts`	OAuth/email provider credentials (NextAuth).
`sessions`	Active user sessions (NextAuth).
`verification_tokens`	Magic-link email verification tokens (NextAuth).

Contributing

Contributions are welcome! Please follow these steps:

Fork the repository and create a feature branch (git checkout -b feat/my-feature).
Make your changes, ensuring the code passes linting (npm run lint).
Commit using a conventional commit message (e.g. feat: add export to BibTeX).
Open a pull request describing what you changed and why.

Code Style

TypeScript is enforced for all Next.js source files.
ESLint is configured via eslint.config.mjs — run npm run lint before committing.
Python code in python-service/ follows PEP 8.

Name		Name	Last commit message	Last commit date
Latest commit History 33 Commits
app		app
components		components
lib		lib
prisma		prisma
public		public
python-service		python-service
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
LICENSE.md		LICENSE.md
README.md		README.md
docker-compose.yml		docker-compose.yml
eslint.config.mjs		eslint.config.mjs
next.config.ts		next.config.ts
package-lock.json		package-lock.json
package.json		package.json
postcss.config.mjs		postcss.config.mjs
prisma.config.ts		prisma.config.ts
tsconfig.json		tsconfig.json

Folders and files

Latest commit

History

Repository files navigation

LitLens

Table of Contents

Features

Architecture

Tech Stack

Frontend

Backend (Node.js)

Python Microservice

Infrastructure

Prerequisites

Quick Start

Environment Variables

Development

Available Scripts

Docker Compose Services

Database Migrations

Python Service (local, without Docker)

Project Structure

API Reference

Authentication

Projects

POST /api/projects

GET /api/projects

GET /api/projects/:id

PATCH /api/projects/:id

DELETE /api/projects/:id

Documents

POST /api/projects/:id/documents

GET /api/projects/:id/documents

GET /api/projects/:id/documents/:docId

PATCH /api/projects/:id/documents/:docId

DELETE /api/projects/:id/documents/:docId

Orientation

POST /api/projects/:id/orient

GET /api/projects/:id/orient

Q&A

POST /api/projects/:id/ask

GET /api/projects/:id/ask

Gap Detection

POST /api/projects/:id/gaps

GET /api/projects/:id/gaps

PATCH /api/projects/:id/gaps/:gapId

Database Schema

Contributing

Code Style

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`POST /api/projects`

`GET /api/projects`

`GET /api/projects/:id`

`PATCH /api/projects/:id`

`DELETE /api/projects/:id`

`POST /api/projects/:id/documents`

`GET /api/projects/:id/documents`

`GET /api/projects/:id/documents/:docId`

`PATCH /api/projects/:id/documents/:docId`

`DELETE /api/projects/:id/documents/:docId`

`POST /api/projects/:id/orient`

`GET /api/projects/:id/orient`

`POST /api/projects/:id/ask`

`GET /api/projects/:id/ask`

`POST /api/projects/:id/gaps`

`GET /api/projects/:id/gaps`

`PATCH /api/projects/:id/gaps/:gapId`