Skip to content

feat: add rocksdb integration and hybrid search implementation#67

Open
ScissorJack-ever wants to merge 6 commits intoAlayaDB-AI:v1.0.0from
ScissorJack-ever:alaya_pr
Open

feat: add rocksdb integration and hybrid search implementation#67
ScissorJack-ever wants to merge 6 commits intoAlayaDB-AI:v1.0.0from
ScissorJack-ever:alaya_pr

Conversation

@ScissorJack-ever
Copy link
Contributor

@ScissorJack-ever ScissorJack-ever commented Feb 3, 2026 •
edited
Loading

Description

  • Added convenient interfaces in rocksdb_storage.hpp for unified scalar data access
  • Integrated scalar data storage into all space types (excluding raw_space) via RocksDB backend
  • Implemented hybrid search functionality in both standard version and RaBitQ-optimized version, with unified query execution pipelines

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Performance improvement
  • Code refactoring
  • Documentation update
  • CI/CD changes

Changes Made

  • Storage Layer: Refactored collection.py to persist all scalar data in RocksDB via the enhanced rocksdb_storage module, replacing previous in-memory or fragmented storage approaches
  • Search Pipeline: Refactored index.hpp to integrate the reranking step directly into the search function, eliminating redundant data transfers and improving query latency

Testing

Describe how you tested your changes:

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
# Commands used to test
make test-cpp && make test-py

Checklist

  • My code follows the project's coding style
  • I have run make lint and fixed any issues
  • I have added tests that prove my fix/feature works
  • All new and existing tests pass (make test)
  • I have updated documentation if needed
  • My commits follow the conventional commits format

@ScissorJack-ever
feat: add RocksDB storage implementation with LZ4/ZSTD compression
f3efab9 
- Implement RocksDBStorage template class with Storage concept
- Configure LZ4 for hot data, ZSTD for cold data compression
- Add comprehensive test suite with 17 test cases covering basic operations, persistence, error handling, and compression
- Improve error handling: throw on data corruption, sync metadata writes
- Update dependencies: add RocksDB, compression libraries, and liburing support
@ScissorJack-ever
feat: add metadata storage support to RaBitQSpace, SQ4Space, and SQ8S…
87c45b1 
…pace

- Integrate RocksDBStorage into space classes for metadata persistence
- Improve RocksDBStorage destructor for enhanced robustness
- Extract RocksDBConfig as a dedicated struct to enable consistent configuration sharing
- Add comprehensive tests to validate RocksDBStorage functionality across all supported spaces
- Add enable-compression option in RocksDBConfig
@ScissorJack-ever
refactor: enhance metadata storage error handling and API consistency
b42b414 
- Add error checks for batch metadata insertion
- Support optional metadata in insert/remove for SQ4Space and SQ8Space
- Simplify RocksDBStorage batch_insert to return bool instead of vector
- Remove unused snappy dependency
@ScissorJack-ever
refactor: refactor graph_search_job.hpp and graph_update_job.hpp to a…
1106ff7 
…lign with intended functionality.

- Remove useless k parameter from search methods, now return ef candidates
- Add ef >= k validation in RaBitQ search methods
- Add explicit rerank step in Python bindings for quantized spaces
- Update all tests to extract topk from ef candidates
- Fix graph_update_job to use sufficient search space size
@ScissorJack-ever
feat: implement hybrid search for SQ4, SQ8, and RaBitQ quantization
b341953 
refactor: relocate reranking logic from index.hpp to graph_search_job.hpp
feat: introduce metadata filtering to integrate with hybrid search
feat: implement scalar data serialization/deserialization for RocksDB storage
feat: expose hybrid search API in collection.py
test: validate hybrid search with diverse filters and quantization types
@ScissorJack-ever
fix: add liburing linkage and improve RaBitQ test robustness
9dab787 
- Link liburing to resolve RocksDB io_uring undefined references
- Add exception handling for RaBitQ tests when AVX512 is unavailable
- Display RocksDB and liburing versions in build summary
@ScissorJack-ever ScissorJack-ever changed the title feat: rocksdb integration and hybrid search implementation feat: add rocksdb integration and hybrid search implementation Feb 3, 2026
@codecov
Copy link

codecov bot commented Feb 3, 2026

Codecov Report

❌ Patch coverage is 88.72367% with 91 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
include/utils/metadata_filter.hpp 67.44% 28 Missing ⚠️
python/src/alayalite/collection.py 84.83% 18 Missing and 9 partials ⚠️
include/storage/rocksdb_storage.hpp 91.06% 21 Missing ⚠️
include/executor/jobs/graph_search_job.hpp 91.52% 5 Missing ⚠️
include/space/sq4_space.hpp 94.64% 3 Missing ⚠️
include/space/sq8_space.hpp 94.64% 3 Missing ⚠️
include/space/rabitq_space.hpp 96.07% 2 Missing ⚠️
include/utils/scalar_data.hpp 97.18% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ScissorJack-ever