[benchmarks] Retrieval benchmark suite — recall@K vs Pinecone, Weaviate, pgvector

[wanruicheng-szyg]

Closes #24

Summary

Adds a self-contained benchmark harness under scripts/bench/ that compares Engram retrieval against Pinecone, Weaviate, and pgvector.

Features

• BEIR dataset loader (nfcorpus, fiqa, scidocs) with synthetic fallback
• EngramClient via local Web API (ingest + query endpoints)
• Stub clients for Pinecone, Weaviate, pgvector
• Metrics: recall@1/5/10, p50/p95 latency
• Markdown report generator

Usage

pip install -r scripts/bench/requirements.txt
uvicorn engram-web.api.main:app --port 8000
python scripts/bench/run.py --engram-only --dataset nfcorpus
python scripts/bench/run.py --all --output report.md

Next steps

• Add Docker Compose service for reproducible hardware
• CI integration with GitHub Actions
• Cost comparison (API pricing vs self-hosted)

[vercel]

@wanruicheng-szyg is attempting to deploy a commit to the praise's projects Team on Vercel.

A member of the Team first needs to authorize it.