Skip to content

Add official library demo catalog - #153

Merged
suguanYang merged 4 commits into
mainfrom
feat/wangbinqi/official-library-catalog
Jun 16, 2026
Merged

suguanYang merged 4 commits into
mainfrom
feat/wangbinqi/official-library-catalog

Conversation

@suguanYang

Copy link
Copy Markdown
Contributor

Summary

  • add an Official Library catalog on top of the existing Knowhere demo-source/materialization flow
  • add staging-parsed demo bundles for 17 new ready library documents plus existing SpaceX, while leaving 4 failed/unsupported URLs planned
  • extend demo-document validation and contract coverage for official-library categories, ready links, and planned items

Ready documents

  • SpaceX S-1, NVIDIA Q1/Q4 materials, NVIDIA FY26 annual report, GOOG 10-K, Meta Q1 materials, Microsoft 2025 annual report, Alibaba FY2026 interim report
  • STEM: Stanford/Jurafsky Transformers, Deep Learning Transformer Network, Probability Theory notes, Statistical Learning notes, Introduction to Statistical Learning Theory

Planned after staging parse failures

  • both Micron static-file URLs: staging job creation timed out repeatedly
  • AIHub Transformers tutorial: staging failed to download the URL
  • Information Theory: 730 pages exceeded direct processing and shard planning failed

Verification

  • Knowhere CLI staging parse batch with provided key (redacted from logs/messages)
  • S3_BUCKET_NAME=test S3_ACCESS_KEY_ID=test S3_SECRET_ACCESS_KEY=test S3_TEMP_PATH=/tmp DATABASE_URL=postgresql://postgres:postgres@127.0.0.1:5432/postgres TMP_PATH=/tmp uv run python apps/api/scripts/validate_demo_documents.py
  • uv run pytest apps/api/tests/contract/test_demo_documents_contract.py -q
  • uv run ruff check apps/api/app/services/demo/official_library_catalog.py apps/api/app/services/demo/source_catalog.py apps/api/scripts/validate_demo_documents.py apps/api/tests/contract/test_demo_documents_contract.py
  • uv run pyright apps/api/app/services/demo/official_library_catalog.py apps/api/app/services/demo/source_catalog.py apps/api/scripts/validate_demo_documents.py apps/api/tests/contract/test_demo_documents_contract.py

@suguanYang
suguanYang merged commit e2a1e9d into main Jun 16, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant