Skip to content

RAG: Create document processing pipeline (internal/rag package) #71

Description

@sre-helmcode

Description

Create internal/rag/ package with:

  • qdrant.go — Qdrant REST client (EnsureCollection, UpsertPoints, Search, DeleteByDocID)
  • embedder.go — Ollama embedding client (Embed, EmbedBatch via POST /api/embed, model: nomic-embed-text)
  • chunker.go — Recursive character text splitting (1000 chars, 200 overlap)
  • parser.go — Document parsing (PDF via ledongthuc/pdf, txt/md, CSV, Excel via excelize, JSON)
  • processor.go — Orchestrator (parse → chunk → embed → upsert)

Acceptance Criteria

  • All parsers handle their formats correctly
  • Chunker produces correct chunks with metadata
  • Embedder calls Ollama API correctly
  • Qdrant client manages collections and points
  • Processor orchestrates the full pipeline
  • Unit tests with mocked Ollama/Qdrant

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions