Knowspan

The living map of your codebase.

A knowledge wiki for source code that grows with Git history. Each wiki page is generated from real commits and tied back to specific files and line ranges, so readers always see how today's docs match today's code. An optional external Q&A agent (powered by pi) lets users ask free-form questions about the repository and turn good answers into proposed wiki edits.

中文版本见 README.zh-CN.md。

Features

• Range-based versioning — every knowledge block carries valid_from_seq / valid_until_seq, so the wiki at any commit is one SQL query away.
• Evidence-first generation — pi-agent-backed planning, page generation, and incremental wiki editing must cite source files / line ranges. The frontend renders source:path#Lx-Ly links inline and opens the matching file in the side panel.
• Per-commit / per-module pages — pages are organised by directory module with a planned, possibly nested table of contents stored in wiki_pages.parent_page_id.
• Background workers — ARQ tasks run reindex / initial generation / incremental updates / per-commit regeneration, with full request + LLM telemetry persisted to request_logs and job_log_entries.
• External Q&A agent — a pi --mode rpc subprocess answers questions in a sandboxed, read-only clone of the repo, uses Knowspan HTTP callback tools for wiki/source context, and persists patch proposals for human review before they touch the wiki.

Repository layout

backend/                FastAPI + SQLAlchemy + Alembic + ARQ
  app/
    api/                Thin HTTP routers (delegates to services)
    services/           Orchestration (git, repo, index, jobs, agent)
    analysis/           File classifier / structure / deps / entry points
    generation/         Agent-backed generation orchestration + persistence
    versioning/         Knowledge-block range engine
    llm/                Provider abstraction + router (Anthropic / OpenAI)
    workers/            ARQ task definitions
    agents/             AgentRuntime ABC + pi RPC subprocess adapter +
                        workspace materializer + session config builder
    models/, schemas/   ORM + Pydantic models
  alembic/versions/     Linear migration chain
  tests/                pytest suites (incl. tests/test_agents/)
frontend/               Next.js 16 (App Router) + Tailwind CSS
backend/pi_extension/   Knowspan pi extension (provider + knowspan_* tools)
docker-compose.yml      Production stack (docker-compose.dev.yml = dev stack)
.env.example            All WIKI_* env vars (copy to .env)

Quick start (Docker)

Requires Docker 24+ with Compose v2. docker-compose.yml is the production stack; see Local development for the hot-reload dev stack.

git clone <this repo>
cd knowspan
cp .env.example .env

# In .env:
#   - set at least one LLM provider key (WIKI_ANTHROPIC_API_KEY recommended)
#   - change WIKI_PI_PROXY_API_KEY and WIKI_POSTGRES_PASSWORD

docker compose up -d --build

The first boot also needs the database schema:

docker compose run --rm backend alembic upgrade head

Once everything is healthy:

Service	URL
Web app + API proxy	http://localhost:3000
API docs (Swagger)	http://localhost:3000/docs

The nginx reverse proxy on WIKI_WEB_PORT (default 3000) is the only published port: Postgres, Redis, the backend, and the frontend stay on the internal compose network, and nginx rejects /api/v1/llm-proxy/... so all LLM traffic originates from the backend. Browser API calls use same-origin /api/v1/....

The API itself has no authentication — deploy on a trusted network or put an authenticating proxy in front of nginx.

Smoke-test the stack:

curl http://localhost:3000/api/v1/repos              # → []
curl -I http://localhost:3000                        # → 200 OK

Then add a repository from the UI (Overview → "Add Repository") and click "Generate Wiki". To use the external Q&A agent, open a repo and use the new "Ask" panel; answers and any auto-detected patch proposals show up under "Proposals".

Local development

Docker dev stack (hot reload, debug logging, DB/API ports published to the host):

docker compose -f docker-compose.dev.yml up -d --build

Or run the processes directly on the host as below.

Backend:

cd backend
uv venv .venv && source .venv/bin/activate
uv pip install -e ".[dev]"
WIKI_DATABASE_URL=postgresql+asyncpg://wiki:wiki@localhost:5432/wiki \
    alembic upgrade head
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

Worker (in a second shell):

cd backend
source .venv/bin/activate
arq app.workers.task_runner.WorkerSettings

Frontend:

cd frontend
npm ci
npm run dev

Bring up just Postgres + Redis via:

docker compose -f docker-compose.dev.yml up -d postgres redis

Local (non-Docker) runs need the pi CLI for wiki generation and Q&A. It is declared as a project-local dependency (Node.js 22.19+) and picked up automatically from backend/node_modules/.bin/pi — make dev installs it on first run, or do it manually:

cd backend && npm install --no-audit --no-fund

Setting WIKI_PI_BIN overrides the resolution (empty = project-local install if present, else pi on the PATH).

The Makefile wraps the common loops:

make dev         # start backend + worker + frontend (uses .env)
make stop-dev

Tests

cd backend
PYTHONPATH=. .venv/bin/pytest                   # full backend suite
PYTHONPATH=. .venv/bin/pytest tests/test_agents # external-agent unit tests

The pi adapter tests (tests/test_agents/test_pi_adapter.py) use a fake pi binary and run offline as part of the normal suite.

Configuration

All runtime knobs live in .env (see .env.example) prefixed with WIKI_. The most important ones:

Variable	Purpose
WIKI_DATABASE_URL	asyncpg DSN for Postgres
WIKI_REDIS_URL	Redis (used by ARQ)
WIKI_ANTHROPIC_API_KEY	Anthropic key (default LLM provider)
WIKI_OPENAI_API_KEY	OpenAI-compatible key (fallback)
WIKI_LLM_CONCURRENCY	Bound on simultaneous LLM calls (default 3)
WIKI_PI_BIN	pi executable; empty = auto (project-local install, then PATH)
WIKI_PI_EXTENSION_PATH	Knowspan pi extension file (empty = backend/pi_extension/knowspan.ts)
WIKI_PI_PROVIDER_ID	pi provider id (defaults to backend proxy knowspan)
WIKI_PI_MODEL_ID	Model id passed to pi through the backend proxy
WIKI_PI_ASK_MODEL_ID	Optional model id for wiki Ask / Q&A sessions (empty = generation model)
WIKI_PI_THINKING_LEVEL	pi thinking level for agent runs (default off)
WIKI_PI_PROXY_BASE_URL	Internal OpenAI-compatible proxy URL exposed by backend
WIKI_PI_PROXY_API_KEY	Shared internal token the pi agent uses to call the backend proxy
WIKI_AGENT_CALLBACK_BASE_URL	URL the agent's tools use to call back into the backend
WIKI_AGENT_WORKSPACES_DIR	Where per-session workspaces are materialised
WIKI_LLM_AUDIT_LOG_DIR	Container directory for JSONL LLM audit files
WIKI_LOG_RETENTION_DAYS	Days to retain request/job/agent/LLM audit logs; set 0 to disable cleanup
WIKI_WEB_PORT	Host port for the Docker nginx web/API proxy (default 3000)
NEXT_ALLOWED_DEV_ORIGINS	Optional comma-separated hosts that may access Next.js dev assets

External agent — what it does

When a user asks a question, AgentService materialises a fresh, read-only copy of the repo at the requested commit (secrets stripped), writes a .knowspan/agent/session.json config, and spawns pi --mode rpc in the workspace to answer using only:

• a planned set of knowspan_* HTTP-callback tools (registered by the Knowspan pi extension), and
• safe built-in read tools (read, grep, find, ls).

pi model calls go through the backend-owned OpenAI-compatible proxy instead of receiving provider keys directly. The proxy uses WIKI_OPENAI_BASE_URL / WIKI_OPENAI_API_KEY for the real upstream and persists audit records in llm_audit_logs and JSONL files under ./dev_data/logs/llm_audit/YYYY-MM-DD.jsonl on the host.

Read-only sessions never expose bash / edit / write to the model — the tool allowlist is enforced by pi's --tools flag per session; only the incremental wiki-edit profile enables shell and file mutation inside the isolated workspace. If the agent decides the answer should be added to the wiki it calls knowspan_propose_patch; the proposal is persisted to wiki_patch_proposals for a human reviewer to approve, reject, or apply via the UI.

License

MIT — see LICENSE.