PaperLoom is an evidence-bounded agentic RAG workbench for reading research-paper PDFs.
The user-facing application is called Folio. It turns a paper into a page-aware Reading Model, lets a tool-using research agent navigate only the papers authorized for the conversation, and keeps the resulting citations reopenable after the model run has ended.
Project site · Documentation · 中文介绍
Most RAG demos optimize the path from a query to plausible text. PaperLoom treats the path from a claim back to its source as part of the product contract:
- PDF content is retained as ordered pages, sections, typed reading elements, locations, and visual assets;
- Java fixes the permission, paper-scope, quota, conversation, and persistence boundaries before research begins;
- Python runs one agent loop through a Java-owned Corpus API inside the authorized Reading Model scope;
- candidate previews are navigation only; citeable evidence exists only after an exact location is read;
- final submission is checked against the evidence ledger before it becomes a product answer;
- historical references remain connected to the paper location that supported the claim.
This repository is also an engineering record. ADRs and selected experiment reports preserve the decisions, failed approaches, costs, and measurements that changed the system.
The live research path is:
ChatHandler
-> ProductReadingConversationService
-> ResearchHarnessTransport
-> RedisResearchHarnessTransport
-> Redis Streams
-> RedisResearchWorker
-> ResearchHarnessService
-> OpenAI Agents SDK Runner
-> Java Corpus API
-> Qdrant Candidate Payload + MySQL-owned validation / structure reads
Java supplies user_id and locked paper IDs. Python calls the Java Corpus API and exposes metadata
discovery, identity resolution, location search, exact reading, optional research guidance, and
validated final submission as tools. It no longer loads every Reading Element into each Harness replica.
The dashed boundary in the Current Runtime diagram makes the repeated Agent Loop explicit: model
request, Function Tool call, protocol validation, corpus-tool result, and the next model request.
Detailed business sequence diagrams are grouped in the Runtime Diagram Catalog: accounts, authorization, paper lifecycle, research chat, the Agent loop, retry/cancel, recovery, PDF evidence, and administration.
Java indexes canonical Current Reading Model locations into Qdrant. sparse-only-v1 runs lexical
BM25 retrieval; sparse-dense-v1 adds MiniMax query embeddings and weighted RRF, with sparse results
as the failure fallback. Qdrant remains a candidate projection. read_paper_content is the only
Agent-visible content tool that returns citeable source_quote_ref values.
PaperLoom does not make raw parser output or a search index its durable paper representation. The Reading Model records:
- model version, readiness, parser provenance, counts, and diagnostics;
- physical pages and readable sections with source spans;
- canonical typed elements such as headings, paragraphs, lists, tables, figures, charts, formulas, footnotes, asides, and code;
- stable page, section, table, and figure locations;
- PDF page screenshots and table, figure, or chart crops.
The Python product adapter first creates lightweight scoped paper-ID shells without Java I/O and hydrates metadata only when the model invokes a paper discovery or identity tool. Java owns the full Reading Model, active lexical index contract, candidate validation, and exact canonical reads. See Reading Model and Agent Tools.
The agent cannot jump directly from a scoped paper ID to a citation. The current authorization ladder is:
Java-authorized paper scope
-> disclosed paper candidate or resolved identity
-> disclosed reading location
-> exact location read
-> Source Quote created
-> final answer validated against known evidence
read_paper_content is the only content tool that returns citeable Source Quotes.
submit_research_answer must be the only tool call in the final step.
| Area | Current responsibility |
|---|---|
| Folio | Vue 3 research workbench, paper selection, progress, conversations, and evidence reopening |
| Java product boundary | Authentication, authorization, locked source scope, quota, cancellation, durable conversations, and reference mappings |
| Python research boundary | Agents SDK loop, tool execution, disclosure state, evidence ledger, citation checks, and final submission |
| Java Corpus plane | Paper authorization, configured Qdrant retrieval, Current Model validation, and exact reads |
| MySQL | Product papers, canonical Reading Models, conversations, and durable reference data |
| Qdrant | Rebuildable sparse or sparse+dense candidate projection keyed by stable location_ref |
| MinIO | Original PDFs, parser artifacts, page screenshots, and crop assets |
| Model provider | MiniMax-M3 used by the Agents SDK runtime through deployment-managed credentials |
Qdrant contributes navigation candidates but never evidence directly. Exact MySQL reads preserve the claim-to-location contract.
PaperLoom records both outcomes and observable research behavior. Optional per-run capture stores ordered model, tool, authorization, evidence, validation, token, latency, and failure events. Golden Cases define required and forbidden papers or evidence, expected facts, claim obligations, outcome, citation policy, trace obligations, and human or judge labels.
That data can support retrieval tuning, tool-policy analysis, provider routing, judge calibration, future dense retriever or reranker training, and teacher-student distillation from strong APIs to a local model. Distillation targets accepted answers and observable tool trajectories, not hidden chain-of-thought. See Evaluation System.
Requirements: Java 17, Maven 3.8+, Node.js 18.20+, pnpm 8.7+, Python 3.11+, Docker Compose v2, and a MinerU Cloud API token for real PDF ingestion.
cp .env.example .env
# Fill the required database, storage, JWT, internal-service, and model credentials.
docker compose --env-file .env -f docs/docker-compose.yaml up -d
python3 -m venv .venv-harness
.venv-harness/bin/pip install -r harness_py/requirements.lock
scripts/paperloom-start-harness.sh start
mvn spring-boot:runNew uploads build the configured Qdrant index automatically. If canonical Current Reading Models were
imported without an index, run the destructive POST /api/v1/admin/retrieval/rebuild-all operation
once after startup. sparse-dense-v1 calls the embedding provider; sparse-only-v1 does not.
In another terminal:
cd frontend
corepack pnpm install
corepack pnpm devFolio is normally available at http://localhost:9527; the backend listens on
http://localhost:8081. The complete setup and first-paper check are in the
Quick Start.
mvn test
.venv-harness/bin/python -m unittest discover -s harness_py/tests
cd frontend
pnpm typecheck
pnpm test:e2e
cd ../site
npm ci
npm run docs:buildPaperLoom is an active engineering project, not a finished hosted service. Its current contract is research-paper PDF ingestion, an inspectable Reading Model, authorized agentic retrieval, and evidence-grounded research chat.
It does not yet claim general-purpose document ingestion, reliable coordinate-level highlighting, automatic citation-graph construction, multimodal retrieval, validated dense-retrieval quality and capacity at large scale, or perfect metadata extraction for every PDF.
- Documentation index
- Architecture overview
- Runtime diagram catalog
- Evaluation system
- Engineering evolution
- Interview materials
- Contributing
- Security policy
PaperLoom is licensed under the Apache License 2.0. Third-party components and assets retain their own licenses; see Third-Party Notices.