read-summarize-papers (`rsp`)

Local AI reading assistant for researchers. Zotero + arXiv → two-tier LLM analysis → Obsidian notes + research map — all from the command line.

What it does

rsp monitors your Zotero library (and optionally auto-fetches from arXiv), runs a two-tier relevance screen, calls an LLM to analyse papers against your personal question pack, and writes structured Obsidian Markdown notes organised by date — with a live research-map canvas on the side.

Zero manual copy-paste. One command per day.

Pipeline

flowchart TD
    subgraph in["📥  Input"]
        A["arXiv\nsources.yaml"] -->|"rsp fetch"| S
        Z["Zotero library\nmanual additions"] -->|"rsp ingest"| S
    end

    S[("SQLite\nstate.sqlite\nstatus: pending")]

    S -->|"rsp process"| KW

    subgraph analysis["🤖  Two-tier LLM analysis"]
        KW{{"Keyword screen\nvs research_profile"}}
        KW -->|"< 2 matches"| SUM["SUMMARY tier\nabstract only\n~10× cheaper"]
        KW -->|"≥ 2 matches"| FULL["FULL tier\nfull text · PDF fallback\nper-question analysis"]
    end

    SUM --> MD1["📄 summary note\npapers/YYYY-MM-DD/KEY.md"]
    FULL --> MD2["📄 structured note\npapers/YYYY-MM-DD/KEY.md"]
    FULL --> GR["📊 research_graph.json"]
    FULL --> CV["🗺️ Research-Relevance.canvas"]

Command map

Group	Commands	Purpose
Ingest	`fetch`, `ingest`	Pull papers into state
Analyse	`process`	Run LLM, write notes
Maintain	`audit`, `duplicates`, `clean`, `organize-papers`	Keep vault tidy
Enrich	`attach-pdfs`, `classify`	Add PDFs, tag Zotero collections
Explore	`search`, `doctor`	Inspect state & debug

Key features

Two-tier processing — summary notes cost ~10× fewer tokens; full notes answer your custom question pack with evidence references
arXiv auto-fetch — define keyword + category sources; rsp fetch runs them and writes directly to Zotero
Incremental & idempotent — content-hash deduplication means re-running is always safe
Duplicate detection — DOI / arXiv ID / title fingerprint; rsp clean merges children and deletes duplicates interactively
Date-organised vault — notes land in papers/YYYY-MM-DD/{zotero_key}.md; rsp organize-papers migrates older flat files
Living research map — full-tier papers grow a JSON Canvas graph in Obsidian
Pluggable LLM backends — local (Ollama, LM Studio, llama.cpp), cloud open-source (Groq, Together.ai, Fireworks, OpenRouter), or cloud proprietary (OpenAI, Anthropic); one config field to switch

Requirements

Python 3.9+
Zotero account + Web API key
Obsidian vault (any local path)
An LLM backend: local (Ollama/LM Studio/llama.cpp — free, no key) or a cloud API key (Groq, Together.ai, Fireworks, OpenRouter, OpenAI, Anthropic)

Setup

1. Install

git clone https://github.com/YOUR_USERNAME/read-summarize-papers
cd read-summarize-papers
pip install -e .

2. Configure

cp config/config.example.yaml config/config.yaml
cp config/question_pack.example.yaml config/question_pack.yaml

Edit config/config.yaml:

vault:
  notes_dir: /path/to/your/obsidian/vault/notes
  paper_notes_subdir: papers      # notes_dir/papers/YYYY-MM-DD/{key}.md
  date_subfolders: true           # organise by processing date

canvas:
  output_path: /path/to/your/obsidian/vault/Research-Relevance.canvas

llm:
  provider: ollama                # see provider table below
  model: llama3.2                 # leave blank to use provider default
  temperature: 0.2
  max_output_tokens: 4096

LLM provider options — set provider: to any of the following:

`provider`	Backend	Key needed?	Default `base_url`
`ollama`	Ollama (local)	No	`http://localhost:11434/v1`
`lm_studio`	LM Studio (local)	No	`http://localhost:1234/v1`
`llamacpp`	llama.cpp server (local)	No	`http://localhost:8080/v1`
`groq`	Groq	`GROQ_API_KEY`	auto
`together`	Together.ai	`TOGETHER_API_KEY`	auto
`fireworks`	Fireworks.ai	`FIREWORKS_API_KEY`	auto
`openrouter`	OpenRouter	`OPENROUTER_API_KEY`	auto
`openai`	OpenAI	`OPENAI_API_KEY`	auto
`anthropic`	Anthropic	`ANTHROPIC_API_KEY`	auto
`openai_compatible`	Any custom endpoint	`OPENAI_API_KEY` or `LLM_API_KEY`	`https://api.openai.com/v1`

Set LLM_API_KEY as a generic fallback when no provider-specific key is configured. Override base_url: in config to point any provider at a different endpoint.

Edit config/question_pack.yaml — set your research_profile and define the questions the LLM should answer for every full-tier paper.

Edit config/sources.yaml — define the arXiv keyword queries to auto-fetch from.

3. Set environment variables

export ZOTERO_LIBRARY_ID="your_numeric_user_id"   # zotero.org/settings/keys
export ZOTERO_API_KEY="your_zotero_api_key"

# Set the key for whichever provider you chose in config.yaml:
export GROQ_API_KEY="your_groq_key"          # provider: groq
export TOGETHER_API_KEY="your_together_key"  # provider: together
export OPENROUTER_API_KEY="your_or_key"      # provider: openrouter
export ANTHROPIC_API_KEY="your_anthropic_key"# provider: anthropic
export OPENAI_API_KEY="your_openai_key"      # provider: openai / openai_compatible
# Local providers (ollama, lm_studio, llamacpp) need no key.
# LLM_API_KEY works as a generic fallback for any provider.

Verify everything is wired up:

rsp doctor --config config/config.yaml

Daily workflow

# Pull new papers from arXiv → Zotero → pending state
rsp fetch --config config/config.yaml

# (Or sync what you added manually in Zotero)
rsp ingest --config config/config.yaml

# Analyse all pending papers and write notes
rsp process --config config/config.yaml --all-pending

Interrupt with Ctrl+C at any time — completed papers are safe and the interrupted one retries next run.

All commands

`rsp doctor`

Validates paths, Zotero API, LLM credentials, and pymupdf. Run first after any config change.

rsp doctor --config config/config.yaml

`rsp fetch`

Queries arXiv sources from config/sources.yaml, adds new papers to Zotero, and marks them pending.

rsp fetch --config config/config.yaml                          # all sources
rsp fetch --config config/config.yaml --source ebsd_kikuchi   # one source
rsp fetch --config config/config.yaml --dry-run               # preview only

config/sources.yaml format:

sources:
  - id: ebsd_kikuchi
    label: "EBSD & Kikuchi patterns"
    provider: arxiv
    priority: 1                                # 1=high, 2=medium, 3=low
    query: "ti:EBSD OR ti:kikuchi"
    categories: [cond-mat.mtrl-sci]
    max_results: 15

`rsp ingest`

Incremental sync from Zotero (based on Last-Modified-Version). Marks new or changed items as pending.

rsp ingest --config config/config.yaml
rsp ingest --config config/config.yaml --no-dedup

`rsp process`

Runs the two-tier pipeline for pending or specified items.

rsp process --config config/config.yaml --all-pending
rsp process --config config/config.yaml --key ABC123XY
rsp process --config config/config.yaml --all-pending --force       # re-run changed items
rsp process --config config/config.yaml --key ABC123XY --force-full # force full analysis

`rsp attach-pdfs`

Downloads PDFs from arXiv and attaches them to Zotero items so you can open them in the Zotero app.

rsp attach-pdfs --config config/config.yaml --dry-run   # preview
rsp attach-pdfs --config config/config.yaml             # attach all missing
rsp attach-pdfs --config config/config.yaml --source ebsd_kikuchi
rsp attach-pdfs --config config/config.yaml --warn-at 70

Stops automatically if Zotero storage is full (free plan: 300 MB).

`rsp audit`

Finds Zotero items that are missing Obsidian notes and reports their state.

rsp audit --config config/config.yaml                  # report only
rsp audit --config config/config.yaml --mark-pending   # queue gaps → then rsp process

Category	Meaning
Not ingested	In Zotero, never seen by `rsp ingest`
Pending	Queued but `rsp process` hasn't run yet
State=done, note missing	Note was moved or deleted
Error	Previous `process` run failed

`rsp duplicates`

Scans for duplicate papers using DOI, arXiv ID, ISBN, and title/author/year fingerprint.

rsp duplicates --config config/config.yaml          # report only
rsp duplicates --config config/config.yaml --mark   # mark in state (skipped by process)

`rsp clean`

Interactively deletes duplicate Zotero items — moves their children to the canonical, deletes the duplicate, removes the vault note.

rsp clean --config config/config.yaml            # interactive: y / s / q per group
rsp clean --config config/config.yaml --dry-run  # preview only

`rsp classify`

Assigns Zotero items to collections based on keyword-match score against sources.yaml queries.

rsp classify --config config/config.yaml --dry-run   # preview
rsp classify --config config/config.yaml
rsp classify --config config/config.yaml --verbose
rsp classify --config config/config.yaml --min-score 2

`rsp organize-papers`

Moves all .md files into papers/YYYY-MM-DD/{zotero_key}.md date subfolders. Useful after migrating from a flat note layout or changing the naming convention. Updates SQLite state with the new paths.

rsp organize-papers --config config/config.yaml --dry-run  # preview
rsp organize-papers --config config/config.yaml            # apply

`rsp search`

Searches your Zotero library by title and shows current processing status.

rsp search "kikuchi patterns" --config config/config.yaml
rsp search "hydrogen reduction" -n 5

ABC123XY  [journalArticle  2024]
  Learning metal microstructural heterogeneity through spatial mapping
  status: done [full]

Outputs

File	Description
`<vault>/papers/YYYY-MM-DD/{zotero_key}.md`	One Obsidian note per paper, grouped by processing date
`<canvas.output_path>`	JSON Canvas research map (updated for every full-tier paper)
`data/research_graph.json`	Persistent graph of papers and question nodes
`data/items/<key>.analysis.json`	Raw LLM JSON response for full-tier papers
`data/state.sqlite`	Processing state, content hashes, tier, duplicate marks, run log

Each Zotero key maps to exactly one note file (path stored in SQLite). Re-processing updates the same file in place.

Note structure

Full-tier note:

---
title: ...
zotero_key: ABC123XY
doi: 10.xxxx/...
pack_version: "2026-01-01"
model: claude-sonnet-4-6
processed_at: "2026-07-17T..."
tier: full
---

## Overview
## Methods
## Variables
## Datasets / data
## Comparison axes
## Claims (paper-stated)
## Limitations
## Relevance to question pack
  ### `q_relevance`
  ### `q_transferable_method`
  ...
## Graph

Summary-tier note:

---
tier: summary
matched_topics: [topic_a]
---

> [!note] Low relevance — 1 topic match(es): topic_a

## Overview
## Methods
## Relevance note

Question pack

config/question_pack.yaml defines your research profile (domain, subtopics, keywords) and the questions the LLM answers for every full-tier paper. Customise it to match your research focus — the questions drive the note structure and the research graph edges.

Priority	Default question	What it extracts
1	`q_problem_fit`	Alignment with your core research question
2	`q_methods_transfer`	Transferable methods / experimental designs
2	`q_variables_evidence`	Key variables, assumptions, evidence chains

License

MIT — see LICENSE.

Name		Name	Last commit message	Last commit date
config		config
src/read_summarize_papers		src/read_summarize_papers
tests		tests
.env.example		.env.example
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
pyproject.toml		pyproject.toml
setup.py		setup.py

Navigation Menu

Repository files navigation

read-summarize-papers (`rsp`)

What it does

Pipeline

Command map

Key features

Requirements

Setup

1. Install

2. Configure

3. Set environment variables

Daily workflow

All commands

`rsp doctor`

`rsp fetch`

`rsp ingest`

`rsp process`

`rsp attach-pdfs`

`rsp audit`

`rsp duplicates`

`rsp clean`

`rsp classify`

`rsp organize-papers`

`rsp search`

Outputs

Note structure

Question pack

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

read-summarize-papers (rsp)

What it does

Pipeline

Command map

Key features

Requirements

Setup

1. Install

2. Configure

3. Set environment variables

Daily workflow

All commands

rsp doctor

rsp fetch

rsp ingest

rsp process

rsp attach-pdfs

rsp audit

rsp duplicates

rsp clean

rsp classify

rsp organize-papers

rsp search

Outputs

Note structure

Question pack

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

read-summarize-papers (`rsp`)

`rsp doctor`

`rsp fetch`

`rsp ingest`

`rsp process`

`rsp attach-pdfs`

`rsp audit`

`rsp duplicates`

`rsp clean`

`rsp classify`

`rsp organize-papers`

`rsp search`