PromptLab — Prompt Management & A/B Testing for LLM Teams

Version, test, and optimize your LLM prompts. Like LaunchDarkly for AI.

---

Features

• Prompt Library — Store, version, and organize all your LLM prompts with full history
• A/B Testing — Create prompt variants, split traffic, measure which performs better
• LLM-as-Judge Evaluations — Run prompts against test datasets and score outputs automatically
• Analytics — Cost, quality score, and latency metrics across all prompts and models
• Serve API — Serve prompts to your application via REST (handles A/B routing automatically)
• Multi-Provider — OpenAI (gpt-4o, gpt-4o-mini) and Anthropic (claude-sonnet-4-6, claude-haiku-4-5)

---

Quick Start

git clone https://github.com/your-org/promptlab
cd promptlab
cp .env.example .env          # add your API keys
docker-compose up             # starts db + backend + frontend

Open http://localhost:3000, register an account, and start building.

---

Architecture

┌─────────────────┐     ┌─────────────────┐     ┌──────────────┐
│  React Frontend │────▶│  FastAPI Backend │────▶│  PostgreSQL  │
│  (TypeScript)   │◀────│  (Python 3.12)  │◀────│  (Database)  │
└─────────────────┘     └────────┬────────┘     └──────────────┘
                                 │
                         ┌───────┴────────┐
                         │  LLM Providers  │
                         │ OpenAI/Anthropic│
                         └────────────────┘

---

API Overview

Method	Path	Description
POST	/api/auth/register	Register user
POST	/api/auth/login	Login, get JWT
GET/POST	/api/projects	List/create projects
GET/POST	/api/projects/{id}/prompts	List/create prompts
POST	/api/prompts/{id}/versions	Create new version
PUT	/api/prompts/{id}/active-version	Set active version
POST	/api/prompts/{id}/test	Test with variables
GET/POST	/api/projects/{id}/experiments	A/B experiments
PUT	/api/experiments/{id}/start	Start experiment
PUT	/api/experiments/{id}/stop	Stop + declare winner
POST	/api/prompts/{id}/evaluate	Run evaluation
GET	/api/serve/{project}/{prompt}/run	Serve prompt (with A/B routing)
GET	/api/analytics/overview	Cost/quality/latency summary

Full OpenAPI docs at http://localhost:8000/docs

---

Tech Stack

Backend: Python 3.12, FastAPI 0.115, SQLAlchemy 2.0 (async), Alembic, Pydantic v2, asyncpg, python-jose, passlib, httpx

Frontend: React 18, TypeScript, Vite, TanStack Router, TanStack Query, Tailwind CSS 3.4, Recharts, Lucide React, Axios, React Hook Form, Zod

Infrastructure: PostgreSQL 16, Docker Compose

---

Development

Running locally without Docker

Backend:

cd backend
pip install -e ".[dev]"
# Set DATABASE_URL env var pointing to a local postgres
alembic upgrade head
uvicorn app.main:app --reload

Frontend:

cd frontend
npm install
npm run dev

Running tests

cd backend
pytest

Adding migrations

docker-compose exec backend alembic revision --autogenerate -m "description"
docker-compose exec backend alembic upgrade head

Seeding sample data

docker-compose exec backend python scripts/seed.py
# Creates demo@promptlab.dev / demo1234

---

Environment Variables

Variable	Default	Description
DB_PASSWORD	promptlab_dev	PostgreSQL password
SECRET_KEY	change-me	JWT signing key (use openssl rand -hex 32)
OPENAI_API_KEY		OpenAI API key (optional)
ANTHROPIC_API_KEY		Anthropic API key (optional)

At least one LLM API key is required for prompt testing, evaluations, and the serve endpoint.

---

License

MIT