Skip to content

Commit c145d67

Browse files
Frankie-XuFrankie-Xu
committed
docs: add missing external-services guide
1 parent 85d5b2b commit c145d67

1 file changed

Lines changed: 299 additions & 0 deletions

File tree

‎docs/external-services.md‎

Lines changed: 299 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,299 @@
1+
# External services
2+
3+
This guide lists the third-party and infrastructure services Knowhere talks
4+
to, and the environment keys that configure them.
5+
6+
Sources:
7+
8+
- `apps/api/.env.example`
9+
- `apps/worker/.env.example`
10+
- [README.md](../README.md) Quick Start and Telemetry sections
11+
12+
Copy those example files to `.env` and fill in the values you need. Do not
13+
commit real credentials, private endpoints, or production URLs.
14+
15+
Most parser and retrieval tuning values have code defaults. Start with the
16+
required external services first, then override model names, provider URLs,
17+
budgets, or concurrency limits only when your deployment needs different
18+
behavior.
19+
20+
## How to read this guide
21+
22+
| Required? | Meaning |
23+
| --- | --- |
24+
| Required for local startup | Listed under "Required for local startup" in `.env.example`. The API and worker need these (or an equivalent local stack) to boot. |
25+
| Required for a feature | Empty is valid until you enable that feature (PDF parse, PPTX convert, billing, webhooks, and so on). |
26+
| Optional override | Commented in `.env.example`. Code already has a default; set the key only to change it. |
27+
28+
**Where keys live.** Unless a row says otherwise, the same key appears in both
29+
`apps/api/.env.example` and `apps/worker/.env.example`. Telemetry keys are
30+
documented on the API side.
31+
32+
**Data leaving the box.** Rows marked **yes** send document bytes, prompts,
33+
images, or metadata to a vendor. Keep that in mind for self-hosted and
34+
air-gapped deployments.
35+
36+
---
37+
38+
## PostgreSQL
39+
40+
| | |
41+
| --- | --- |
42+
| **Purpose** | Primary store for users, documents, chunks, graph, retrieval, and billing state. |
43+
| **Required?** | Required for local startup. |
44+
| **Data leaving the box?** | No, unless you point `DATABASE_URL` at a hosted database. Local-dev uses the Compose Postgres from `./deploy/local-dev/start-dev.sh`. |
45+
46+
| Key | Notes |
47+
| --- | --- |
48+
| `DATABASE_URL` | SQLAlchemy URL (`postgresql+asyncpg://...`). |
49+
| `DB_SSL_MODE` | `disable`, `allow`, `prefer`, `require`, `verify-ca`, or `verify-full`. |
50+
| `DB_SSL_CERT` | Optional client certificate path. |
51+
| `DB_SSL_KEY` | Optional client key path. |
52+
| `DB_SSL_ROOT_CERT` | Optional CA path. |
53+
54+
---
55+
56+
## Redis and Celery
57+
58+
| | |
59+
| --- | --- |
60+
| **Purpose** | Cache, rate limiting, job coordination, and the Celery broker / result backend. |
61+
| **Required?** | Required for local startup. |
62+
| **Data leaving the box?** | No, unless you point Redis at a hosted instance. Local-dev uses Compose Redis. |
63+
64+
| Key | Notes |
65+
| --- | --- |
66+
| `REDIS_HOST` | Redis hostname. |
67+
| `REDIS_PORT` | Redis port. |
68+
| `REDIS_PASSWORD` | Leave empty for unauthenticated local Redis. |
69+
| `REDIS_DATABASE` | Logical database index. |
70+
| `CELERY_REDIS_URL` | Full Redis URL for Celery broker, result backend, and RedBeat. |
71+
| `BROKER_POOL_LIMIT` | Celery broker connection-pool size. |
72+
| `RATE_LIMIT_ENABLED` | API rate limiting; not a vendor key. |
73+
74+
---
75+
76+
## S3-compatible object storage
77+
78+
| | |
79+
| --- | --- |
80+
| **Purpose** | Upload originals, store parse artifacts, and (when configured) hold result bundles. Local-dev uses LocalStack. |
81+
| **Required?** | Required for local startup. |
82+
| **Data leaving the box?** | Document files are written to the configured buckets. LocalStack on the local-dev stack stays on the machine. A cloud S3/MinIO/OSS endpoint stores objects with that provider. |
83+
84+
| Key | Notes |
85+
| --- | --- |
86+
| `S3_TYPE` | Storage backend. `.env.example` sets `s3` for local-dev LocalStack. |
87+
| `S3_BUCKET_NAME` | Upload / source bucket. |
88+
| `S3_RESULTS_BUCKET` | Results bucket (also used by billing). |
89+
| `S3_ACCESS_KEY_ID` | Access key. Pair with `S3_SECRET_ACCESS_KEY`. |
90+
| `S3_SECRET_ACCESS_KEY` | Secret key. |
91+
| `S3_ENDPOINT_URL` | Custom endpoint (LocalStack, MinIO, or other S3-compatible APIs). |
92+
| `S3_PRIVATE_DOMAIN` | Private asset domain. |
93+
| `S3_TEMP_PATH` | Local temp path used during storage operations. |
94+
| `S3_REGION` | Region name; may be empty for MinIO. |
95+
| `S3_USE_SSL` | TLS for the storage client. |
96+
| `S3_ADDRESSING_STYLE` | `auto`, `path`, or `virtual`. |
97+
| `S3_WEBHOOK_AUTH_TOKEN` | Shared secret for S3/MinIO event webhooks. |
98+
| `OSS_ENDPOINT` | Aliyun OSS endpoint when `S3_TYPE=oss`. Required for that backend. |
99+
| `OSS_EVENT_CALLBACK_KEY` | OSS upload-callback signing key. Feature-specific. |
100+
| `OSS_EVENT_VERIFY_SIGNATURE` | Verify OSS event signatures. |
101+
102+
---
103+
104+
## DeepSeek / OpenAI-compatible LLM
105+
106+
| | |
107+
| --- | --- |
108+
| **Purpose** | Text and table summarization, heading hierarchy, and other OpenAI-compatible chat calls. README: configure at least one of `DS_KEY`, `ALI_API_KEYS`, `GPT_API_KEY`, or `GLM_API_KEY`. |
109+
| **Required?** | Required for local startup: **at least one** provider key. URLs and model names have code defaults. |
110+
| **Data leaving the box?** | **Yes.** Chunk text, table HTML, and prompts are sent to the provider you enable. |
111+
112+
| Key | Notes |
113+
| --- | --- |
114+
| `DS_KEY` | DeepSeek API key. |
115+
| `DS_URL` | Optional. Default is documented in `.env.example`. |
116+
| `GPT_API_KEY` | OpenAI API key. |
117+
| `GLM_API_KEY` | Zhipu GLM API key. |
118+
| `GLM_URL` | Optional GLM base URL. |
119+
| `ARK_API_KEY` | Volcengine Ark API key. |
120+
| `ARK_URL` | Optional Ark URL. |
121+
| `NORMOL_MODEL` | Optional text-model override (summaries and general LLM calls). |
122+
| `HIERARCHY_LLM_MODEL` | Optional heading-hierarchy model; falls back to `NORMOL_MODEL`. |
123+
124+
README also lists swapping OpenAI, DashScope, Zhipu, or Volcengine through
125+
these variables.
126+
127+
---
128+
129+
## Qwen / VLM (DashScope)
130+
131+
| | |
132+
| --- | --- |
133+
| **Purpose** | Image summaries, OCR, PDF coarse classification, atlas routing, and other vision calls. Default model name in `.env.example` is `qwen3.6-flash`. |
134+
| **Required?** | Required for image / OCR / atlas / image-aware retrieval. Typical key: `ALI_API_KEYS`. |
135+
| **Data leaving the box?** | **Yes.** Extracted images and vision prompts are sent to the configured VLM endpoint (DashScope by default). |
136+
137+
| Key | Notes |
138+
| --- | --- |
139+
| `ALI_API_KEYS` | Aliyun DashScope key pool (JSON array, comma/newline list, or `token_id=api_key`). |
140+
| `ALI_URL` | Optional DashScope-compatible base URL. |
141+
| `IMAGE_MODEL` | Optional default VLM name. |
142+
| `IMAGE_MODEL_MAX` | Optional higher-capability VLM name. |
143+
144+
---
145+
146+
## Cursor (agent_explore harness)
147+
148+
| | |
149+
| --- | --- |
150+
| **Purpose** | Default agentic retrieval path (`agent_explore`). The harness talks to Cursor when `AGENT_EXPLORE_HARNESS` is `cursor_sdk`. |
151+
| **Required?** | Required for the default agentic route when the harness is `cursor_sdk`: set `CURSOR_API_KEY`. Set `use_agentic=false` on the query to stay on classic map-unit BM25 and skip this vendor. The `openai` harness reuses the OpenAI-compatible LLM keys above instead of Cursor. |
152+
| **Data leaving the box?** | **Yes, when `cursor_sdk` is used.** Query text and corpus-tool traffic go to Cursor. |
153+
154+
| Key | Notes |
155+
| --- | --- |
156+
| `AGENT_EXPLORE_HARNESS` | Optional. Commented in `.env.example`. Default is `cursor_sdk`. The other documented value is `openai`. |
157+
| `CURSOR_API_KEY` | Required when the harness is `cursor_sdk`. Commented in `.env.example`. |
158+
| `AGENT_EXPLORE_CURSOR_MODEL` | Optional. Commented in `.env.example`; example default is `composer-2.5`. |
159+
160+
---
161+
162+
## MinerU
163+
164+
| | |
165+
| --- | --- |
166+
| **Purpose** | Default PDF parser. Knowhere sends PDFs (or reusable S3 URLs) to MinerU, then rebuilds hierarchy from the Markdown / layout output. |
167+
| **Required?** | Required for PDF parsing (`MINERU_API_KEYS`). |
168+
| **Data leaving the box?** | **Yes.** PDF bytes or presigned object URLs go to the MinerU API. |
169+
170+
| Key | Notes |
171+
| --- | --- |
172+
| `MINERU_API_KEYS` | MinerU API key pool (same pool syntax as `ALI_API_KEYS`). |
173+
| `MINERU_URL` | Optional API base URL. |
174+
| `MINERU_TOKEN_RPM_LIMIT` | Optional per-token RPM limit. |
175+
| `MINERU_TOKEN_DAILY_LIMIT` | Optional per-token daily limit. |
176+
| `MINERU_TOKEN_COOLDOWN_SECONDS` | Optional cooldown after rate limiting. |
177+
| `MINERU_UPLOAD_MODE_ENABLED` | Optional. Default prefers S3 URL mode when the source object is reusable. Set true only when MinerU cannot fetch your storage URLs. |
178+
| `MINERU_SHARD_CONCURRENCY` | Local cap on concurrent MinerU calls for oversized PDF shards. |
179+
180+
Related local PDF limits in the same `.env.example` files (not vendor keys):
181+
`MAX_PDF_PAGE_LIMIT`, `OVERSIZED_PDF_SHARD_ENABLED`, `OVERSIZED_PDF_SOFT_LIMIT`,
182+
`PDF_PROFILE_TOC_ENABLED`.
183+
184+
---
185+
186+
## iLoveAPI
187+
188+
| | |
189+
| --- | --- |
190+
| **Purpose** | PPTX → PDF conversion before the PDF / MinerU pipeline. |
191+
| **Required?** | Required for PPTX parsing. |
192+
| **Data leaving the box?** | **Yes.** Presentation files are uploaded to iLoveAPI / iLovePDF. |
193+
194+
| Key | Notes |
195+
| --- | --- |
196+
| `ILOVEAPI_PUBLIC_KEY` | Project public key. |
197+
| `ILOVEAPI_SECRET_KEY` | Project secret key. |
198+
| `ILOVEAPI_BASE_URL` | Optional API base URL. |
199+
| `ILOVEAPI_TIMEOUT` | Optional request timeout in seconds. |
200+
201+
---
202+
203+
## PostHog telemetry
204+
205+
| | |
206+
| --- | --- |
207+
| **Purpose** | Anonymous self-hosted product telemetry (install liveness, usage aggregates, client/document mix). Schema and allowlists: [ADR-0004](adr/0004-anonymous-self-hosted-telemetry.md). |
208+
| **Required?** | Default-on. Not required to run the product. Opt out with `TELEMETRY_ENABLED=false`. Keys are in `apps/api/.env.example` only. |
209+
| **Data leaving the box?** | **Yes, if enabled** — only allowlisted anonymous events. Events must not include filenames, prompts, emails, IPs, geo, or document content. |
210+
211+
| Key | Notes |
212+
| --- | --- |
213+
| `TELEMETRY_ENABLED` | `true` by default. Set `false` to disable. |
214+
| `TELEMETRY_POSTHOG_HOST` | Optional ingestion host. |
215+
| `TELEMETRY_POSTHOG_PROJECT_KEY` | Optional project token. Commented in `apps/api/.env.example`. |
216+
| `TELEMETRY_INSTALLATION_ID` | Optional operator-provided install id. |
217+
| `TELEMETRY_INSTALLATION_ID_PATH` | File used to persist a generated install id. |
218+
| `TELEMETRY_BATCH_SIZE` | Optional event batch size. |
219+
| `TELEMETRY_REQUEST_TIMEOUT_SECONDS` | Optional outbound timeout. |
220+
| `TELEMETRY_DEPLOYMENT_MODE` | Optional label on events (default `self_hosted`). |
221+
| `TELEMETRY_AGGREGATE_INTERVAL_SECONDS` | Optional aggregate snapshot interval. |
222+
223+
---
224+
225+
## Stripe
226+
227+
| | |
228+
| --- | --- |
229+
| **Purpose** | Credits / checkout billing for hosted product flows. |
230+
| **Required?** | Required when `BILLING_ENABLED=true` and you accept payments. Self-hosted and API-only local work can leave Stripe empty (`BILLING_ENABLED` is `false` in the worker example). |
231+
| **Data leaving the box?** | **Yes, when enabled.** Customer and payment objects go to Stripe. Document content is not the billing payload. |
232+
233+
| Key | Notes |
234+
| --- | --- |
235+
| `BILLING_ENABLED` | Master switch for Stripe/credits billing. |
236+
| `STRIPE_SECRET_KEY` | Stripe secret key. |
237+
| `STRIPE_WEBHOOK_SECRET` | Stripe webhook signing secret. |
238+
| `FREE_PLAN_INITIAL_CREDITS` | API-only. Starting credits for the free plan. |
239+
| `FRONTEND_URL` | API-only. Checkout success/cancel return URL. |
240+
241+
---
242+
243+
## Other optional integrations
244+
245+
These are listed in `.env.example` under observability, webhooks, or
246+
analytics. They are not required to parse documents locally.
247+
248+
### Logfire
249+
250+
| | |
251+
| --- | --- |
252+
| **Purpose** | Optional distributed tracing / logging. |
253+
| **Required?** | Optional. Leave `LOGFIRE_TOKEN` empty unless you use Logfire. |
254+
| **Data leaving the box?** | **Yes, if a token is set.** Traces and log metadata go to Logfire. |
255+
256+
| Key | Notes |
257+
| --- | --- |
258+
| `LOGFIRE_TOKEN` | Logfire API token. |
259+
260+
### QStash (outbound webhooks)
261+
262+
| | |
263+
| --- | --- |
264+
| **Purpose** | Deliver async webhooks through Upstash QStash, with signed callbacks into the API. |
265+
| **Required?** | Required for QStash-backed webhook delivery. |
266+
| **Data leaving the box?** | **Yes, if enabled.** Webhook payloads go to QStash, then to your public callback URL. |
267+
268+
| Key | Notes |
269+
| --- | --- |
270+
| `QSTASH_TOKEN` | QStash API token. |
271+
| `QSTASH_CALLBACK_BASE_URL` | Public API base URL used to build QStash callback paths. |
272+
| `QSTASH_MAX_RETRIES` | Delivery retry count. |
273+
| `QSTASH_CURRENT_SIGNING_KEY` | Optional current signing key. |
274+
| `QSTASH_NEXT_SIGNING_KEY` | Optional next signing key (rotation). |
275+
| `WEBHOOK_MASTER_KEY` | Local webhook encryption key, not a QStash credential. |
276+
277+
### Moesif
278+
279+
| | |
280+
| --- | --- |
281+
| **Purpose** | Optional API analytics. |
282+
| **Required?** | Optional. |
283+
| **Data leaving the box?** | **Yes, if `MOESIF_APPLICATION_ID` is set.** API request metadata goes to Moesif. |
284+
285+
| Key | Notes |
286+
| --- | --- |
287+
| `MOESIF_APPLICATION_ID` | Moesif application id. |
288+
289+
---
290+
291+
## Keys that are not vendor credentials
292+
293+
The same `.env.example` files also contain local runtime knobs (file limits,
294+
parser schema, entity types, standalone API mode, dashboard URL, and
295+
`RETRIEVAL_POSTGRES_FTS_CANDIDATE_LIMIT`). They are not external services.
296+
See the comments in those files.
297+
298+
HTML and HTM ingestion is implemented in the worker
299+
(`formats/html` + `SUPPORTED_EXTENSIONS`). It does not add a separate vendor.

0 commit comments

Comments
 (0)