Skip to content
BranchesTags

Latest commit

 

History

85 Commits

Folders and files

NameName
Last commit message
Last commit date
backend
backend
 
 
data
data
 
 
docs
docs
 
 
eval
eval
 
 
frontend
frontend
 
 
java-backend
java-backend
 
 
.env.example
.env.example
 
 
.gitignore
.gitignore
 
 
AGENTS.md
AGENTS.md
 
 
CLAUDE.md
CLAUDE.md
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
docker-compose.yml
docker-compose.yml
 
 

Repository files navigation

Cite Scope

Agentic research assistant for cited paper Q&A

Quick Start | Architecture | English | MIT License

Python FastAPI Vue LangGraph License

Cite Scope 是一个面向学术论文问答的 Agentic RAG 系统。它既能通过 arXiv ID/URL 拉取官方 metadata 和 PDF，也能处理 PDF、Office、网页/文本、表格和图片等本地异构文件；统一解析入库后用 Qdrant + MySQL 管理来源、片段和会话，再用 LangGraph Agent 进行意图分析、动态规划、多源检索、资料充分性评估、自我反思和带引用回答生成。

这个项目的目标不是做一个最短链路的聊天壳，而是展示一个可解释、可调试、可扩展的论文研究助手：用户能看到系统检索了什么、为什么补充检索、哪些论文被引用、回答可信度为什么高或低。

Highlights

  • Lean agentic workflow: 8 个 LangGraph 控制/检查点节点覆盖 guard -> analyze -> plan -> executor -> evidence_gate -> synthesis -> groundedness -> finalize；内部保留原 13 个节点的业务职责及细粒度 stage / StepTrace，并通过 re-planning 做有界补充检索。
  • Adaptive retrieval plan: Planner 根据问题复杂度生成不同检索计划，而不是固定跑一条 RAG pipeline。
  • Hybrid retrieval: Qdrant dense vector search 与 BM25 sparse ranking 融合，支持 oversampling、alpha 权重和检索缓存。
  • Heterogeneous ingestion: arXiv 与 PDF、DOCX、PPTX、HTML、Markdown/TXT、CSV/XLSX、图片共用 parse -> normalize -> chunk -> index -> persist，并保留 text/table/image_ocr 模态和 page/slide/sheet locator。
  • Self-verification: sufficiency 阶段调用 evaluate_docs 检查证据充分性，groundedness 检查引用、完整性与逻辑；失败后按预算触发补充检索或重新生成。
  • Transparent execution: 前端通过 SSE 展示每一步耗时、参数、结果摘要、检索片段和调试详情。
  • Corpus overview: 新对话和论文库页面可以展示当前 RAG 语料库的主题分布、代表论文和建议问题。
  • Cited answers: 回答保留论文来源卡片和 citation popover，便于回到 arXiv / PDF 证据。
  • Operational basics: SQLite LangGraph checkpoint、会话历史、反馈接口、异步 arXiv 导入任务、API key 鉴权和限流中间件。
  • Optional web search: Tavily 作为可选补充检索，未配置或网络失败时会降级，不阻断本地 RAG。

Product Surface

Area What it does
Chat 提问、流式回答、查看执行步骤、引用来源和调试详情
Papers 浏览已入库论文、检索论文、查看语料库主题 overview
Uploads 输入 arXiv ID/URL，或批量拖放异构文件；查看解析、OCR、分块和索引进度
Settings 查看后端连接、模型和基础配置状态
Feedback 对回答标记有帮助 / 需改进，为后续评估闭环保留数据

Architecture

User query + chat history
        |
        v
  guard
  - empty/oversize/injection checks
        |
        v
  analyze
  - intent + conservative complexity router
        |
        v
  plan
  - planner/re-planner + deterministic source route
  - local-first source policy
        |
        v
  executor loop
  - retrieve_local
  - retrieve_arxiv
  - search_web
  - query_rewrite
  - get_paper_detail
  - get_paper_chunks
        |
        v
  evidence_gate
  - dedupe/rerank/context budget + sufficiency
    | insufficient + budget
    +-----------------------> plan(re-planner) -> executor
    |
    v
  synthesis (streaming)
        |
        v
  groundedness
    | re-retrieve ---------> plan(re-planner) -> executor
    | re-generate ---------> synthesis
    v
  finalize
  - citation gate + presentation -> SSE/UI

Backend

backend/app/
├── agent/          LangGraph state, graph, checkpoint, streaming, nodes
├── tools/          Retrieval, web search, evaluation, paper lookup tools
├── routers/        FastAPI routes for chat, papers, uploads, ingest, feedback
├── services/       Heterogeneous document parsing, unified ingestion, hybrid retrieval
├── db/             MySQL and Qdrant clients
├── models/         SQLAlchemy ORM models
├── schemas/        Pydantic API contracts
└── middleware/     Rate limit, API key auth, request context

Frontend

frontend/src/
├── components/     Chat, answer cards, source cards, corpus overview
├── composables/    SSE and chat orchestration
├── stores/         Pinia state for chat, conversations, theme
├── api/            Typed API clients
├── views/          Chat, papers, uploads, settings
└── utils/          Markdown, durations, thinking-step detail mapping

Storage

Storage Purpose
MySQL 8 Paper metadata, chunks, conversations, chat history, upload jobs, feedback
Qdrant Dense vectors for paper chunks
BM25 cache Sparse retrieval over chunk text for hybrid ranking
SQLite checkpoint LangGraph thread checkpoint state
Local data/ arXiv PDFs, uploaded source files, metadata JSON and checkpoint files

Quick Start

1. Start infrastructure

docker compose up -d mysql qdrant
cp .env.example .env

Edit .env and fill at least:

LLM_API_KEY=...
EMBEDDING_API_KEY=...

Image and scanned-PDF OCR uses the local Tesseract binary. On macOS:

brew install tesseract tesseract-lang

Current defaults use:

LLM_MODEL=MiniMax-M2.7
LLM_API_BASE=https://api.minimax.chat/v1
EMBEDDING_MODEL=BAAI/bge-m3
EMBEDDING_API_BASE=https://api.siliconflow.cn/v1

Optional:

TAVILY_API_KEY=...        # web search supplement
API_AUTH_ENABLED=true     # enable API key auth for deployment
API_KEYS=your-key-1,...

2. Run backend

cd backend
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

Health check:

curl http://localhost:8000/health

/health 是进程存活检查，不会调用外部 LLM/Embedding 服务，因此不能证明 API Key 有效。完整就绪验证还应发起一次实际问答，并确认响应中的 degraded 为 false；若为 true，先检查对应供应商凭据与额度。

3. Build a paper corpus

Small recent arXiv corpus:

cd backend
python scripts/download_arxiv.py --limit 50
python scripts/ingest.py

Focused AI landmark corpus:

cd backend
python scripts/curate_ai_landmark_corpus.py --target 500 --workers 6
python scripts/ingest.py --force

The curation script writes data/metadata_filtered.json and downloads available PDFs into data/pdfs/. Papers without accessible PDFs are recorded in data/raw_metadata/ai_landmark_skipped.json and are not ingested until a PDF is supplied.

4. Run frontend

cd frontend
npm install
npm run dev

Open:

http://localhost:5173

5. Docker full stack

cp .env.example .env
# fill API keys first
docker compose up -d --build

修改 .env 后 Compose 会重建受影响的容器；修改后端或前端源码后必须保留 --build，否则可能继续运行旧镜像。若只需更新后端，可执行 docker compose up -d --build backend。

Frontend:

http://localhost:8080

Backend:

http://localhost:8000

Configuration

Important .env groups:

Group Keys
LLM LLM_MODEL, LLM_API_BASE, LLM_API_KEY, optional PLANNER_MODEL, REFLECTION_MODEL
Embedding EMBEDDING_MODEL, EMBEDDING_API_BASE, EMBEDDING_API_KEY
Retrieval RETRIEVAL_K, FINAL_CONTEXT_K, HYBRID_ALPHA, HYBRID_OVERSAMPLE, cache settings
Agent AGENT_MAX_PLAN_STEPS, AGENT_MAX_REFLECTIONS, AGENT_CHECKPOINT_ENABLED, AGENT_CHECKPOINT_PATH
Search TAVILY_API_KEY, ARXIV_MAX_RESULTS
Safety RATE_LIMIT_ENABLED, API_AUTH_ENABLED, API_KEYS, auth exempt paths
Ingestion UPLOAD_DIR, INGEST_MAX_FILE_MB, INGEST_MAX_ARCHIVE_UNCOMPRESSED_MB, OCR settings
Data DATA_DIR, PDF_DIR, METADATA_JSON

See .env.example for the full list.

API Overview

Endpoint Purpose
GET /health backend health check
POST /chat synchronous chat response
POST /chat/stream SSE streaming chat response
GET /conversations list conversations
GET /conversations/{id}/messages load persisted messages
GET /papers list/search ingested papers
GET /papers/overview corpus topic overview
POST /upload/arxiv queue arXiv ID/URL import and background ingest
POST /upload/files queue 1–20 heterogeneous local files for background ingest
GET /upload/jobs list upload jobs
POST /feedback store answer feedback
POST /ingest admin ingestion endpoint

FastAPI docs are available at:

http://localhost:8000/docs

Validation

Backend:

backend/.venv/bin/python -m pytest -q

Frontend:

cd frontend
npm test
npm run build

The test suite mocks LLM and database boundaries. It should not call real LLM, embedding, Tavily, arXiv or Qdrant services.

Documentation

File Description
AGENTS.md Architecture, design philosophy, agent flow and contributor conventions
CLAUDE.md Short agent context for future coding sessions
eval/README.md Evaluation framework and retrieval tuning notes
docs/cite-scope-java-architecture-overview.md High-level architecture review and roadmap status
docs/cite-scope-defect-verification.md Defect verification notes
docs/cite-scope-update-plan.md Iteration plan
docs/cite-scope-execution-acceptance.md Acceptance notes

Roadmap

  • Stronger feedback consumption: connect answer feedback to evaluation/admin review.
  • Richer corpus management: deduplication, import/export, and source-level quality controls.
  • Production hardening: multi-instance rate limiting, stricter auth defaults, migrations and deployment profiles.
  • Evaluation dashboards: trend reports for retrieval quality, citation faithfulness and answer confidence.

Notes

  • This repository does not commit PDFs, vector data, database files or API keys.
  • You need your own LLM and embedding API keys before the full RAG path can run.
  • Tavily web search is optional. Local retrieval remains the primary path.
  • API authentication is configurable and disabled by default for local development. Enable it before public deployment.
  • Some legacy internal identifiers for existing databases, collections, metrics, packages and browser storage are intentionally retained for compatibility.

English

Cite Scope is an Agentic RAG system for academic document Q&A. It imports arXiv papers and heterogeneous local files, stores provenance-aware chunks in MySQL, indexes embeddings in Qdrant, and uses a LangGraph agent to plan retrieval, evaluate evidence, synthesize cited answers, and self-reflect before returning the final response.

Why this project exists

Cite Scope is designed as a portfolio-grade, explainable research assistant. Instead of hiding retrieval behind a black box, it exposes execution steps, retrieved chunks, source cards, citation details, confidence reasons and debug traces in the UI.

Key capabilities

  • An 8-node LangGraph control graph that preserves the former 13 node responsibilities and fine-grained stage/trace observability.
  • Hybrid dense + BM25 retrieval with tunable ranking parameters.
  • Optional Tavily web search for evidence supplementation.
  • SSE streaming for live answer tokens and execution-step updates.
  • Persistent conversations, chat history and LangGraph checkpoints.
  • A shared background-ingestion state machine for arXiv plus PDF, DOCX, PPTX, HTML, text, spreadsheet, and image sources.
  • Corpus overview for topic buckets, representative papers and suggested questions.
  • Vue 3 frontend with source cards, citation popovers and debug panels.

Tech stack

Layer Technology
Agent LangGraph
Backend FastAPI, SQLAlchemy, Pydantic
Frontend Vue 3, Vite, Tailwind CSS, Pinia
LLM MiniMax M2.7 through an OpenAI-compatible API
Embedding SiliconFlow BAAI/bge-m3 by default
Vector DB Qdrant
SQL DB MySQL 8
Streaming Server-Sent Events

Quick start

docker compose up -d mysql qdrant
cp .env.example .env
# fill LLM_API_KEY and EMBEDDING_API_KEY

Backend:

cd backend
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

Ingest sample papers:

cd backend
python scripts/download_arxiv.py --limit 50
python scripts/ingest.py

Frontend:

cd frontend
npm install
npm run dev

Open http://localhost:5173.

License

Cite Scope is released under the MIT License.

About

Local-first arXiv RAG: Ollama (gemma4:e4b + bge-m3) + MySQL/SQLite + Chroma + Vue3

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages