[benchmarks] Retrieval benchmark suite — recall@K vs Pinecone, Weaviate, pgvector

[Dipraise1]

Context

ROADMAP Phase 3 calls for public retrieval benchmarks to back up Engram's positioning: recall@K, latency, and cost vs Pinecone, Weaviate, and pgvector on public datasets.

Tasks

• Pick 2–3 public embedding datasets (e.g. BEIR subsets)
• Reproducible harness (scripts/bench/) that runs the same queries against Engram and each baseline
• Report recall@1/5/10, p50/p95 latency, and storage overhead (note Engram's (k,n) erasure coding)
• Publish results in docs/benchmarks.md

Listed as an open bounty in ROADMAP.md.

[KHHH2312]

Resolved by #25

/claim #24
Contact handle for payout: @OxaamE on X (https://x.com/OxaamE)
Payout wallet address: 0x43991A9dC8Ddf492eab6E55685644c2cb9B001D2

[HamzaKhanBUIC]

/claim I'll handle this bounty immediately!

[wanruicheng-szyg]

I would like to work on this benchmark suite. I have experience with Python benchmark frameworks, vector DBs (Pinecone, pgvector), and the BEIR evaluation suite. I can build a reproducible harness with recall@K and latency metrics against all three baselines.

Plan:

• Use BEIR nfcorpus/fiqa/scidocs subsets
• Dockerized harness with consistent hardware
• Automated report generation (JSON + markdown)
• CI-ready with GitHub Actions

Happy to discuss scope and timeline.

[zjwjing]

I would like to work on this benchmark suite. I will create a reproducible harness that compares Engram against Pinecone, Weaviate, and pgvector on public datasets.

Plan:

1. Select 2-3 public embedding datasets (BEIR subsets)
2. Create scripts/bench/ harness
3. Run recall@1/5/10 benchmarks
4. Measure p50/p95 latency
5. Document results in docs/benchmarks.md

Wallet: 0x8971bC40f06e90015C5e92d247c96BfB65C24Cc7

[kaising-openclaw1]

I've submitted PR #51 implementing the retrieval benchmark suite as described in this issue. The harness supports Engram, Pinecone, Weaviate, and pgvector on BEIR datasets (NQ, HotpotQA, FiQA), reporting recall@1/5/10, p50/p95/p99 latency, and storage overhead.

PR: #51

Looking forward to feedback!