add CICD - #14
Merged
Merged
add CICD#14
Conversation
- Add deploy.sh for one-click deployment with Python 3.10 virtual environment - Add activate.sh for quick environment activation - Add build_all.sh for unified algorithm building - Add install_packages.sh for package management - Add GitHub Actions workflow for automated testing - Remove redundant CI workflows (ci.yml, cmake.yml) - Update .gitignore to exclude virtual environments and build artifacts - Add ALGORITHM_DEPLOYMENT.md with comprehensive deployment guide - Add isolated Docker testing for algorithm builds - Configure workflow to run on main, main-dev, and test branches
- Upgrade actions/cache from v3 to v4 - Upgrade actions/upload-artifact from v3 to v4 - Resolves deprecation warning in GitHub Actions
The cache was causing issues because: - Virtual environment wasn't created when cache was restored - PyCANDYAlgo and torch modules were not found - Deploy script needs to run fresh each time to ensure proper setup
- test-deployment now runs complete ./deploy.sh without skipping - test-quick-build only skips algorithm build, installs system deps - Ensures CI tests the full user experience from scratch
- build_all.sh now properly checks build.sh exit code - build.sh handles missing torch.utils.cmake_prefix_path gracefully - CMake configuration errors will now properly fail the build - Prevents silent build failures that lead to missing .so files
- test-deployment already includes all functionality of test-quick-build - Reduces CI runtime and resource usage - Only keep test-deployment (full build test) and lint-and-format
Problem: Default torch installation includes CUDA which causes CMake to fail on systems without CUDA (like GitHub Actions runners). Solution: - deploy.sh now explicitly installs torch CPU version first - Requirements.txt updated with note about CPU-only installation - CMake will find CPU-only torch and configure successfully This ensures deployment works on systems without CUDA/GPU.
- deploy.sh now installs Intel oneAPI MKL from Intel repos - build.sh sets MKLROOT and LD_LIBRARY_PATH for Puck CMake - Ensures all PyCANDY dependencies (DiskANN, Puck) can build Puck requires MKL libraries which are not available by default. This adds the Intel oneAPI repository and installs mkl-devel package.
This directory was incorrectly ignored by /DiskANN/ pattern in .gitignore. DiskANN is required for PyCANDY build and must be in the repository. Without it, CMake fails with 'diskann_s target not found' error.
- CMakeLists.txt: Improve library linking and error handling - build.sh: Add MKL environment setup and better torch path detection - vsag_hnsw: Update configuration parameters
Issues fixed: 1. Add libgoogle-glog-dev (glog) - required by PyCANDY 2. Add libfmt-dev (fmt) - required by DiskANN 3. Add libnuma-dev (numa) - required by SPTAG 4. Fix MKL header path - add CPATH for mkl.h 5. Fix libgomp conflict - prioritize system libgomp over torch's version These dependencies were missing from apt-get install, causing: - CMake Error: MKL_H set to NOTFOUND - CMake Error: Findfmt.cmake not found - WARNING: Numa library not found - WARNING: glog not found - libgomp.so.1 hidden by torch/lib
Resolved issues: 1. MKL_H NOTFOUND - Added Intel oneAPI setvars.sh sourcing and CMAKE_PREFIX_PATH 2. fmt library not found - Installed libfmt-dev package 3. Additional warnings: - numa library - Installed libnuma-dev package - glog not found - Installed libgoogle-glog-dev package - libgomp conflict - Set LD_LIBRARY_PATH to prioritize system library All system dependencies now properly installed and environment configured.
- Add libunwind-dev to dependency check list - Add libunwind-devel for yum package manager - Add libunwind for brew package manager - Fixes build error: libgoogle-glog-dev requires libunwind-dev
- Add spdlog to dependency check list - Add spdlog-devel for yum package manager - Add spdlog for brew package manager - Fixes GTI build error: spdlog/spdlog.h not found
- Add google-perftools (tcmalloc) to dependency check list - Add gperftools-devel for yum package manager - Add gperftools for brew package manager - Fixes GTI link error: cannot find -ltcmalloc_minimal
- Add libaio to dependency check list - Add libaio-devel for yum package manager - Skip libaio for macOS (not needed) - Fixes compile error: libaio.h: No such file or directory
Add all missing dependencies required by algorithms: - libgflags-dev: gflags library - liblapack-dev, libblas-dev, libopenblas-dev: Linear algebra libraries - libboost-all-dev: Boost C++ libraries - swig: Interface compiler - libhdf5-dev: HDF5 file format library - libomp-dev: OpenMP library - libtbb-dev: Intel Threading Building Blocks - libarchive-dev: Archive library - libcurl4-openssl-dev: cURL library - python3-dev: Python development headers - libeigen3-dev: Eigen3 template library - zlib1g-dev: Compression library - libssl-dev: SSL/TLS library This ensures all dependencies from Dockerfile are automatically installed.
The error 'dynamic module does not define module export function' occurs because DiskANN's module.cpp defines its own PYBIND11_MODULE (_diskannpy), which conflicts with PyCANDYAlgo's module definition. Instead of using GLOB with all cpp files, explicitly list only the needed DiskANN Python binding source files: - builder.cpp - dynamic_memory_index.cpp - static_disk_index.cpp - static_memory_index.cpp This excludes module.cpp and diskann_bindings.cpp which contain conflicting PYBIND11_MODULE definitions, allowing PyCANDYAlgo to properly export diskannpy as a submodule.
Changes: 1. Make glog a required library (not optional) 2. Link glog before DiskANN since DiskANN depends on it 3. Use pkg-config to find glog properly with all dependencies 4. Add explicit comment about linking order requirement This fixes the 'undefined symbol: _ZN6google15LogMessageFatalC1EPKci' error by ensuring proper library linking order and making glog mandatory.
1. glog detection improvements: - Try find_package(glog) first - Fallback to pkg-config with proper LINK_LIBRARIES - Try direct library search in common paths - Final fallback to -lglog linker flag - Remove FATAL_ERROR to allow build to proceed 2. Fix third-party library install permissions: - Set CMAKE_INSTALL_PREFIX to local build/install directory - Prevents permission denied errors when installing to /usr/local - Applies to GTI, IP-DiskANN, and PLSH 3. Add ALGORITHMS_IMPL_DIR variable at script start - Used for local install prefix paths - Create build/install directory upfront This should fix both the 'glog not found' CMake error and the 'Permission denied' install errors.
The error 'undefined symbol: _ZN5faiss20IndexHNSWSQOptimizedC1E...' occurs because faiss symbols are not properly resolved during linking. Changes: 1. Move faiss to the beginning of link libraries (before TORCH) 2. Add faiss again at the end of link list to resolve circular deps 3. This ensures all faiss symbols are properly linked Linux linker processes libraries in order, so we need faiss both at the start (for direct usage) and at the end (for circular deps).
The undefined symbol error for IndexHNSWSQOptimized occurs because IndexHNSWOptimized.cpp was not included in the faiss build. This file contains the implementation of IndexHNSWSQOptimized class which is used by CANDY algorithms but wasn't being compiled into the faiss library. Adding it to FAISS_SRC in faiss/CMakeLists.txt resolves the undefined symbol: _ZN5faiss20IndexHNSWSQOptimizedC1E... error.
- Add check for Makefile existence before compilation - Re-run CMake configuration if Makefile is missing - Return to algorithms_impl directory to ensure correct path - Fix issue where make fails with 'No makefile found' error
修复 PyCANDYAlgo 导入失败问题： - 在导入测试中添加 algorithms_impl 目录到 sys.path - 确保 Python 能找到 PyCANDYAlgo.so 文件
优化措施： 1. 添加 CMake 和 pip 缓存，避免重复构建和下载 2. 设置并行编译环境变量（MAKEFLAGS, CMAKE_BUILD_PARALLEL_LEVEL） 3. 保留 build 缓存，只清理 .so 文件 4. 所有分支都跳过 VSAG 构建（通过 CI 环境变量） 分支策略： - test 分支：快速测试（排除性能测试） - main/main-dev 分支：完整测试 预期提升： - 首次构建：~30-40 分钟（跳过 VSAG 节省 15-20 分钟） - 缓存命中后：~5-10 分钟（test 分支） - 缓存命中后：~10-15 分钟（main 分支）
- 使用 python3 替代 python 确保版本正确 - 简化导入测试为单行命令避免 YAML 缩进问题 - 在 algorithms_impl 目录中执行导入测试 - 添加 Python 版本和路径确认步骤
1. 修复 IP-DiskANN/GTI/PLSH 安装权限错误 - 使用本地安装目录而非系统目录 (/usr/local/lib) - 添加 CMAKE_INSTALL_PREFIX 参数指向 build/install - make install 失败不中断构建流程 2. 修复 PyCANDYAlgo 模块导入错误 (PyInit_PyCANDYAlgo 未定义) - pybind11_add_module 添加 MODULE 参数 - 设置符号可见性为 default (-fvisibility=default) - 移除过于严格的链接器标志 (--no-undefined) - 确保 Python 初始化函数正确导出
1. 修正 pybind11_add_module 调用方式 - MODULE 参数单独一行，确保正确解析 - 显式设置 OUTPUT_NAME 属性 2. 添加符号导出链接选项 - 使用 -Wl,--export-dynamic 确保符号可见 - 保留 -fvisibility=default 编译选项 3. 增强 CI/CD 诊断 - 添加 nm 命令检查 PyInit 符号 - 显示导出符号列表帮助调试 - deploy.sh 中也添加符号检查
- 添加 IndexHNSWFlatOptimized pybind11 绑定，支持 reorder_gorder() - 简化 deploy.sh，仅保留 PyCANDYAlgo 构建 - 简化 CI/CD workflow - 更新 faiss_HNSW_Optimized 算法适配新接口 - 修复 compute_gt.py 和 export_results.py 的 import 问题 - 添加 simple runbook 到重要实验列表
- GTI: 先构建 n2 依赖，在 bindings 目录构建 - IP-DiskANN: 添加 -DPYBIND=ON 参数 - 所有模块: 添加 pybind11 cmake 路径解决找不到 pybind11 的问题
- 移除 n2 依赖构建（目录不存在） - 先构建 GTI 主项目生成库和头文件 - 再构建 Python bindings
- 添加 libspdlog-dev 系统依赖（n2 库需要） - 先构建 n2 库，再构建 GTI 主项目，最后构建 bindings - 按照 GTI README 的正确构建顺序
主要修改： 1. 修复 n2 库的 spdlog API 兼容性问题 - 构建时自动添加缺失的头文件 spdlog/sinks/stdout_color_sinks.h - 兼容 spdlog 1.9.2+ 版本的 stdout_color_mt API 2. 修复 ABI 不匹配问题 - n2 库编译时使用 -D_GLIBCXX_USE_CXX11_ABI=0 以匹配 GTI 3. 优化构建流程 - deploy.sh: 简化 GTI 构建流程，一次构建完成主项目和 bindings - deploy.sh: 添加 VSAG (pyvsag) 构建支持 - build_all.sh: 应用相同的修复逻辑 4. CI/CD 改进 - 添加 VSAG 缓存和验证步骤 - 添加 VSAG 构建日志上传 测试结果： - ✅ 本地构建成功 - ✅ CI 环境模拟测试通过 - ✅ 所有模块可正常导入
## VSAG (pyvsag) Fixes - Add MKL environment variable configuration in deploy.sh - Set LD_LIBRARY_PATH for MKL libraries before building - Pass MKLROOT to CMake configuration - Resolves: ImportError: libmkl_intel_lp64.so.2 ## GTI (gti_wrapper) Fixes - Add libgoogle-perftools-dev to system dependencies - Make tcmalloc optional in GTI CMakeLists.txt - Build only gti_wrapper target (Python bindings) by default - GTI executable (which needs tcmalloc) is optional - Resolves: cannot find -ltcmalloc_minimal ## Documentation - Add CICD_FIXES.md with detailed explanation - Document all changes and testing procedures Tested in CI/CD environment and verified both modules build and import successfully.
## Problem - VSAG builds successfully but fails to import after environment reload - Error: libmkl_intel_lp64.so.2: cannot open shared object file - LD_LIBRARY_PATH was set during build but not persisted ## Solution 1. Modify virtual environment's activate script to auto-set MKL path 2. Add MKL library path configuration when creating venv 3. Ensure MKL environment is available on every venv activation 4. Add MKL path check before module import tests ## Changes - deploy.sh: Add MKL path to venv activate script (step 2) - deploy.sh: Ensure MKL env before tests (step 9) - CICD_FIXES.md: Update documentation with persistence solution This ensures pyvsag works correctly even after exiting and re-entering the virtual environment.
- registry.py: 新增 get_all_algorithm_param_combinations() 获取所有参数组合 - run_benchmark.py: 重构主循环，支持遍历多个参数组合并分别保存结果 - config.yaml: 简化vsag_hnsw配置，统一参数组合格式
- export_results.py: 添加--params, --list-params, --all-params选项 - 支持参数化目录结构的结果导出 - 每个参数目录保存详细结果(*_final_results.csv) - 根目录生成汇总表(algorithm_summary.csv) - vsag_hnsw config.yaml: 启用全部三种prefetch模式
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.