A RAG (Retrieval-Augmented Generation) system designed for competitive analysis of LLM observability platforms.
This project implements a PDF Knowledge Explorer that allows you to:
- Upload and index PDF documents
- Ask natural language questions about the content
- Get AI-generated answers with citations
- Compare how 9+ different observability platforms capture and display traces
- Measure actual SDK overhead for each observability provider
- Compare latency impact with statistical analysis (avg, median, P95, P99)
- Identify the fastest providers for your use case
- Export results for decision-making
- Auto-test each provider for supported features
- Automatically fill comparison matrix based on actual behavior
- Save hours of manual testing
- Get confidence scores for each detection
- Start testing immediately with pre-generated sample PDF
- Includes comprehensive content about LLM observability
- 8 sample queries covering different scenarios
- No need to find and upload your own documents
- New criteria categories: Debugging & RCA, Security, Optimization, Agentic Logic
- Auto-fill from capability detection
- Export to JSON/Markdown
- Visual rankings and charts
See IMPROVEMENTS.md for detailed documentation.
| # | Platform | SDK | Integration Method |
|---|---|---|---|
| 1 | LangSmith | langsmith |
RunTree API / @traceable decorator |
| 2 | Langfuse | langfuse |
@observe decorator + trace() API |
| 3 | Arize Phoenix | arize-phoenix |
OpenTelemetry instrumentor |
| 4 | Opik | opik |
@track decorator |
| 5 | Braintrust | braintrust |
@traced decorator + init_logger() |
| 6 | Laminar | lmnr |
@observe decorator |
| 7 | AgentOps | agentops |
agentops.init() + auto-instrumentation |
| 8 | Evidently AI | evidently |
ML monitoring (no real-time tracing) |
| 9 | Pydantic Logfire | logfire |
logfire.instrument_openai() + @span |
graph TD
User[User] -->|Query| QA[Query Analyzer]
QA -->|Analyzed Query| RE[Retriever]
subgraph "Hybrid Retrieval"
RE -->|Vector Search| VS[ChromaDB]
RE -->|Keyword Search| BM25[BM25 Index]
VS -->|Results| RRF[RRF Merge]
BM25 -->|Results| RRF
end
RRF -->|Ranked Chunks| RR[Reranker]
RR -->|Reranked Context| AG[Reasoning Engine]
subgraph "Agentic Logic"
AG -->|Loop| Tools[Tools: Calc, Search]
Tools -->|Output| AG
end
AG -->|Augmented Context| GEN[Generator]
GEN -->|Response| GE[Graph Extractor]
GE -->|Final Output| User
| Step | What it does | Span Type | What we test |
|---|---|---|---|
| Query Analyzer | Determines intent, extracts entities | LLM | Prompt/completion logging, classification |
| Retriever | Vector search in ChromaDB | RETRIEVER | Retrieved docs, scores, metadata |
| Reranker | Sorts chunks by relevance | LLM | Scoring, batch processing |
| Generator | Generates answer with citations | LLM | Streaming, token count, cost |
| Graph Extractor | Extracts concept relationships | LLM | Structured output, JSON parsing |
# Clone the repository
git clone <repo-url>
cd pdf-knowledge-rag
# Create virtual environment
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install dependencies
pip install -e .- Copy the example environment file:
cp .env.example .env- Add your API keys:
# Required
OPENAI_API_KEY=sk-...
# Select which providers to use
OBSERVABILITY_PROVIDERS=langsmith,langfuse
# Add keys for selected providers
LANGCHAIN_API_KEY=ls-...
LANGFUSE_PUBLIC_KEY=pk-...
LANGFUSE_SECRET_KEY=sk-...
# ... etcstreamlit run app.pyfrom src.pipeline.traced_orchestrator import TracedRAGOrchestrator
from pathlib import Path
# Initialize
orchestrator = TracedRAGOrchestrator()
# Ingest PDFs
doc, chunks = orchestrator.ingest_pdf(Path("paper.pdf"))
print(f"Ingested {chunks} chunks")
# Query
result = orchestrator.query("What is the attention mechanism?")
print(result.response.answer)
# View trace URLs
for provider, url in orchestrator.obs_manager.get_trace_urls(trace).items():
print(f"{provider}: {url}")from scenarios import ScenarioRunner
runner = ScenarioRunner()
results = runner.run_all()
runner.export_results("results.json")
runner.print_summary()- Simple RAG - Basic query to test fundamental tracing
- Multi-hop - Cross-document retrieval with parallel spans
- Long Context - Large payload handling and cost tracking
- Streaming - Real-time trace updates and TTFT
- Error Handling - Error capture and trace status
- Evaluation - Batch evaluation with ground truth
The comparison evaluates each platform across 7 categories:
- Time to first trace
- Lines of code for integration
- Auto-instrumentation support
- Documentation quality
- Nested/parallel spans
- Metadata support
- Input/output capture
- Streaming support
- Error handling
- Token counting
- Cost calculation
- Prompt/completion display
- Multi-provider support
- Document display
- Relevance scores
- Chunk previews
- Source metadata
- Built-in evals
- Custom eval functions
- Human feedback UI
- Dataset management
- A/B comparison
- Dashboards
- Search/filtering
- Trace comparison
- Alerting
- UI quality
- Self-hosted option
- Pricing
- Enterprise features
- Debugging UX - How quickly can you find problems in traces?
- Retrieval visibility - How are retrieved chunks displayed?
- Cost tracking - Automatic vs manual? How accurate?
- Eval workflow - How easy to run evaluations?
- Performance overhead - SDK latency impact?
- Cold start - Time from signup to useful trace?
- No good diff between traces
- Poor retrieval quality visualization
- Hard to connect production issues with prompts
- No easy "replay" for problematic traces
- Weak CI/CD integration for regression testing
pdf-knowledge-rag/
app.py # Streamlit UI
pyproject.toml # Project configuration
.env.example # Environment template
src/
config.py # Configuration management
models.py # Data models
pipeline/ # RAG pipeline components
ingestion.py # PDF parsing & chunking
query_analyzer.py # Intent detection
retriever.py # Vector search
reranker.py # LLM reranking
generator.py # Response generation
graph_extractor.py# Concept extraction
orchestrator.py # Pipeline coordination
traced_orchestrator.py # With observability
observability/ # Provider integrations
base.py # Abstract interfaces
manager.py # Multi-provider coordination
providers/ # Individual implementations
evaluations/ # Comparison tools
comparison_matrix.py
scenarios/ # Test scenario definitions
data/ # PDF storage & ChromaDB
results/ # Exported results
screenshots/ # UI screenshots for comparison
Contributions are welcome! See CONTRIBUTING.md for details.
This project is licensed under the MIT License - see the LICENSE file for details.
See SECURITY.md for reporting vulnerabilities.
- Code of Conduct: We follow the Contributor Covenant.
- Issues: Use the issue tracker for bugs and features.
- Discussions: (Optional) Join our community discussions.
Developed for research and comparison of AI observability platforms.