LanceDB Benchmark

This repository contains benchmarks for evaluating AI models on LanceDB, a serverless, developer-friendly, open-source vector database for AI. You can view the evaluation reports at https://kweizh.github.io/lancedb-benchmark/.

What is LanceDB

LanceDB is an open-source, serverless vector database designed for AI workloads. Built on top of the Lance columnar data format, it provides blazing-fast vector search, full-text search, and hybrid search capabilities with zero-copy data loading and direct S3 storage integration. Key features tailored for modern AI and agentic workloads include:

• Multi-Modal Support: Store and query vector embeddings, metadata, and raw multi-modal data (like images and audio) in a single, unified table.
• Hybrid Search & Rerankers: Out-of-the-box support for combining vector search and full-text search (FTS) with advanced rerankers (like RRF, Linear Combination, and Cohere).
• Zero-Copy & Native Arrow: Direct integration with Apache Arrow, PyArrow, Pandas, and Pydantic for ultra-fast, zero-copy data manipulation and streaming.
• Serverless-First Storage: Store your database locally, on S3, or on GCS with native versioning, time-travel, and automatic index optimization.
• Rich Ecosystem Integration: Direct support for embedding registries (OpenAI, HuggingFace, Cohere, etc.) and native vector store integrations with LangChain and LlamaIndex.

About this Leaderboard

This leaderboard is generated by Zealt to evaluate how effectively different AI agents and Large Language Models (LLMs) can utilize the LanceDB APIs, Python SDK, and ecosystem integrations. As AI coding assistants become more prevalent, it's crucial to understand their proficiency with modern, vector-native database primitives. This benchmark provides a standardized environment to measure an agent's capability in handling end-to-end LanceDB development and maintenance tasks, including:

• table lifecycle and schema evolution (e.g., column additions, deletions, nested structures)
• vector and scalar index management (e.g., IVF_PQ index creation, BITMAP/BTREE scalar filters)
• native full-text search (FTS) and hybrid query construction
• advanced reranking and relevance feedback (e.g., Rocchio, Cohere, RRF)
• multi-tenant namespace isolation
• automated database maintenance and optimization (e.g., compaction, index monitoring)
• integration with LangChain and LlamaIndex and others.

How to Contribute

We welcome contributions from the community to make this benchmark more comprehensive! Here are a few ways you can help:

1. Add New Benchmark Tasks

You can create new evaluation scenarios in the tasks/ directory. Good tasks should test specific aspects of LanceDB, such as advanced hybrid search weights, custom schema evolution, temporal window queries, per-column compression settings, multi-modal blob storage, or framework-specific vector store configurations. Each task is defined in its own directory and must follow a specific structure compatible with the Harbor framework:

• instruction.md: The prompt and instructions provided to the AI agent.
• task.toml: Configuration for the task (metadata, timeouts, etc.).
• environment/Dockerfile: The Docker environment definition where the task and evaluations are executed.
• bootstrap/: Contains files to set up the initial state before the agent starts (e.g., task.json, test_initial_state.py).
• tests/: Contains the evaluation scripts to verify the agent's solution (e.g., test.sh, test_final_state.py).

2. Submit New Evaluation Results

Help us expand the leaderboard by running evaluations for new models or agents and submitting the results in the jobs/ directory. This benchmark is evaluated using the Harbor framework. You can run the evaluation using the Harbor CLI:

harbor run \
  --agent codex \
  --model "gpt-5.2-codex" \
  --env daytona \
  --path ./tasks \
  --n-attempts 1 \
  --max-retries 5 \
  --n-concurrent 5 \
  --retry-include RuntimeError \
  --retry-include DaytonaError \
  --retry-include AgentTimeoutError

Note: Before starting, ensure you have set the necessary environment variables for your chosen agent (including any API keys or credentials needed, such as OpenAI or Cohere API keys). Evaluations can be run locally with Docker (default) or using Daytona.io. When running with Daytona, please note that Daytona blocks some network access for tier 1 and tier 2 users; see Daytona network limits if you encounter network issues.

3. Improve the Visualization Site

The leaderboard visualization is a Next.js application located in the site/ directory. Contributions to improve the UI, add new metrics, or enhance the data presentation are highly appreciated.

---

Generated by Zealt