Skip to content

README.md

Latest commit

 

History

History
215 lines (162 loc) · 8.19 KB

README.md

File metadata and controls

215 lines (162 loc) · 8.19 KB
Raw

spanIQ

CI License: Apache 2.0 Python 3.10+

Deterministic LLM evaluation and production monitoring. $0/trace. No LLM judge.

1,000 test cases × 3 metrics → $0.00 and 4 seconds instead of $34.20 and 9,000 API calls.

Get your first signal in 2 minutes

pip install spaniq
from spaniq.core.test_case import LLMTestCase
from spaniq.core.evaluate import evaluate
from spaniq.metrics.semantic_similarity import SemanticSimilarityMetric

tc = LLMTestCase(
    input="What is the capital of France?",
    actual_output="Paris is the capital of France.",
    expected_output="Paris",
)
result = evaluate([tc], [SemanticSimilarityMetric()])
# Deterministic. Same score every run. $0.00.

Or with OTel (zero code changes to your app):

pip install "spaniq[otel]"
spaniq collect-otel --baseline my_baseline
# point your OTel exporter to localhost:4317

Then see your data:

pip install "spaniq[dashboard]"
spaniq dashboard
# opens http://localhost:8501

The benchmark table

Accuracy at catching bad outputs on a labeled 100-row QA set (positive class = bad; thresholds calibrated on a held-out fold). Live run, 2026-07-10.

Tool Precision Recall F1 ROC-AUC Cost / trace Deterministic
Groq LLM judge 1.000 1.000 1.000 1.000 paid usually, at temp 0
deepeval (G-Eval) 1.000 0.833 0.909 0.971 paid usually, at temp 0
spanIQ 0.806 0.967 0.879 0.857 $0.00 ✅ by construction
langfuse-style 0.612 1.000 0.760 0.525 paid usually, at temp 0

The honest read: on subtle, lexically-close wrong answers, a real LLM judge beats spanIQ's embedding metric on accuracy — and we say so. spanIQ's edge is being deterministic and $0/trace by construction, and staying useful where judges default to passing everything (on the summarization / RAG sets the judges collapse to F1 0.667 while spanIQ scores 1.0). They are complementary. Full tables, the fairness rule, and a per-item audit trail: BENCHMARK_ACCURACY.md.

Run this yourself: spaniq benchmark --tool spaniq --runs 5 --metric accuracy Full suite (Groq key): spaniq benchmark --tool spaniq,groq,deepeval,ragas,langfuse --runs 5 --metric accuracy

Architecture

                 ┌─────────────────────────────────────┐
                 │         Streamlit Dashboard          │  ← V4
                 │  reads SQLite, renders interactive   │
                 └──────────────┬──────────────────────┘
                                │ reads
     ┌──────────────────────────┼──────────────────────┐
     │                   SQLite DB                      │
     │  baselines │ timeline │ alerts │ components      │
     └──────────────────────────┬──────────────────────┘
                                │ writes
┌───────────────────────────────┼───────────────────────┐
│              V1-V3 Pipeline (untouched)                │
│  Metrics → Monitor → TimelineStore → Attribution      │
└───────────────────────────────┬───────────────────────┘
                                │ feeds
┌───────────┬───────────┬───────┴───┬───────────────────┐
│  File     │ Langfuse  │   SDK     │  OTelCollector    │  ← V4
│ Collector │ Collector │ Collector │  OTLP/gRPC+HTTP   │
└───────────┴───────────┴───────────┴───────────────────┘

V4 adds one collector at the bottom and one visualization layer at the top. The middle is untouched.

Features

V1 — Deterministic evaluation

  • SemanticSimilarityMetric — cosine similarity with sentence-transformers
  • OutputStabilityMetric — distributional consistency against a reference sample
  • ConsistencyMetric — self-consistency across paraphrased prompts
  • ResponseDriftMetric — statistical drift detection (PSI, KS, JS divergence)

V2 — Production monitoring

  • Monitor — stream traces from file, Langfuse, or SDK; score against baseline
  • TimelineStore — SQLite-backed score time series with trend analysis
  • AlertEngine — consecutive-failure alerts with JSONL + SQLite logging
  • Collectors: FileCollector, LangfuseCollector, SDKCollector

V3 — Root-cause attribution

  • PELT changepoint detection per component
  • CUSUM online alarms
  • Cascade attribution: identifies root-cause component and downstream cascades
  • spaniq attribute CLI command with terminal report and PNG export

V4 — Ecosystem entry (this release)

  • OTelCollector — OTLP/gRPC + HTTP receiver; auto-maps GenAI semconv spans
  • Streamlit dashboard — 4 pages: Overview, Drift Timeline, Attribution, Alert Log
  • Accuracy benchmark — compare spanIQ vs Groq/deepeval/ragas/langfuse on precision/recall/F1 at catching bad outputs (plus cost and determinism)
  • Quickstart overhaul — 2-line first-signal experience
  • Generic span support — spaniq.* attributes for non-GenAI apps

Install

# Core eval + monitoring
pip install spaniq

# OTel receiver
pip install "spaniq[otel]"

# Local dashboard
pip install "spaniq[dashboard]"

# Benchmark suite
pip install "spaniq[benchmark]"

# Groq baseline collection
pip install "spaniq[groq]"

# Everything
pip install "spaniq[all]"

CLI reference

# Baseline management
spaniq baseline collect --name my_baseline --prompt "..." --source groq --n 30
spaniq baseline list
spaniq baseline show my_baseline

# Production monitoring
spaniq monitor run --baseline my_baseline --source file --path traces.jsonl
spaniq collect-otel --baseline my_baseline   # NEW: OTel receiver

# Timeline inspection
spaniq timeline show --metric ResponseDriftMetric
spaniq timeline summary --metric SemanticSimilarityMetric
spaniq timeline export --metric ResponseDriftMetric --output chart.png

# Pipeline attribution (V3)
spaniq pipeline run --name my_pipeline --path traces.jsonl
spaniq attribute --pipeline my_pipeline --last 500

# Dashboard (V4)
spaniq dashboard                             # NEW
spaniq dashboard --db spaniq.db --port 8501

# Benchmark (V4)
spaniq benchmark --tool spaniq --runs 5 --metric accuracy      # NEW
spaniq benchmark --tool spaniq,groq,deepeval,ragas,langfuse --runs 3 --metric accuracy

Comparison

Feature spanIQ deepeval ragas Phoenix Langfuse
Deterministic by construction (no LLM call) ✅ ❌ ❌ ❌ ❌
$0 per eval ✅ ❌ ❌ ❌ ❌
Production monitoring ✅ ❌ ❌ ✅ ✅
Changepoint attribution ✅ ❌ ❌ ❌ ❌
OTel integration ✅ ❌ ❌ ✅ ✅
Local dashboard ✅ ❌ ❌ ✅ ✅
No cloud required ✅ ✅ ✅ ❌ ❌
No API key to evaluate ✅ ❌ ❌ ❌ ❌

Documentation

License

Apache 2.0