Skip to content

[benchmarks] Add retrieval benchmark suite for Engram vs Pinecone, Weaviate, pgvector#48

Open
zjwjing wants to merge 8 commits into
Dipraise1:mainfrom
zjwjing:benchmark-suite
Open

[benchmarks] Add retrieval benchmark suite for Engram vs Pinecone, Weaviate, pgvector#48
zjwjing wants to merge 8 commits into
Dipraise1:mainfrom
zjwjing:benchmark-suite

Conversation

@zjwjing

@zjwjing zjwjing commented Jun 18, 2026

Copy link
Copy Markdown

Description

Adds a retrieval benchmark suite to compare Engram against Pinecone, Weaviate, and pgvector.

Closes #24

Changes

New Files

  • scripts/bench/benchmark.py - Benchmark harness
  • docs/benchmarks.md - Report template

Features

  • Recall@1, Recall@5, Recall@10 metrics
  • p50/p95 latency measurement
  • Support for multiple vector databases
  • Markdown report generation
  • Reproducible test harness

Datasets

  • MSMARCO
  • NFCorpus
  • NQ (Natural Questions)

Wallet Address

ETH: 0x8971bC40f06e90015C5e92d247c96BfB65C24Cc7

@vercel

vercel Bot commented Jun 18, 2026

Copy link
Copy Markdown

@zjwjing is attempting to deploy a commit to the praise's projects Team on Vercel.

A member of the Team first needs to authorize it.

@zjwjing zjwjing left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi maintainers,

I have completed all the requested changes and the PR is ready for review.

Please let me know if you need any additional modifications.

Thank you!

@zjwjing

zjwjing commented Jun 23, 2026

Copy link
Copy Markdown
Author

Hi @Dipraise1 — following up on the retrieval benchmark suite PR. The benchmark harness, metrics (Recall@1/5/10, p50/p95 latency), and report generation are all implemented and ready for review. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[benchmarks] Retrieval benchmark suite — recall@K vs Pinecone, Weaviate, pgvector

1 participant