From 0bfa0067367c577aefded8c5985bc5f2bc23ea33 Mon Sep 17 00:00:00 2001 From: Gordon YUEN Date: Wed, 29 Jul 2026 17:50:22 +0800 Subject: [PATCH 1/5] docs: add AGENTS.md, agent skills config, ADR 0001, and upload helper script - AGENTS.md: compact instruction file for OpenCode sessions covering repo purpose, verification, build flow, env layering, runtime architecture, conventions, and agent skills config - docs/agents/: issue tracker (local markdown under .scratch/), triage labels (canonical defaults), domain docs (single-context layout) - docs/adr/0001-local-mineru-mode.md: ADR for local MinerU mode via forked image patch, moved from repo root - .gitignore: add .scratch/ for local issue tracking - upload-pdf.sh: helper script to upload a PDF via the API from start to confirm-upload --- .gitignore | 1 + AGENTS.md | 92 ++++++++++++++++++++++++++++++ docs/adr/0001-local-mineru-mode.md | 87 ++++++++++++++++++++++++++++ docs/agents/domain.md | 53 +++++++++++++++++ docs/agents/issue-tracker.md | 19 ++++++ docs/agents/triage-labels.md | 15 +++++ upload-pdf.sh | 42 ++++++++++++++ 7 files changed, 309 insertions(+) create mode 100644 AGENTS.md create mode 100644 docs/adr/0001-local-mineru-mode.md create mode 100644 docs/agents/domain.md create mode 100644 docs/agents/issue-tracker.md create mode 100644 docs/agents/triage-labels.md create mode 100755 upload-pdf.sh diff --git a/.gitignore b/.gitignore index ca708e9..f458068 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,4 @@ logs/ __pycache__/ *.pyc +.scratch/ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..d8e4b27 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,92 @@ +# AGENTS.md + +## What this repo is (and is not) + +This repo **only packages** Knowhere for self-hosted Docker Compose deployment. It contains **no application code**. The API, worker, and dashboard source live in two separate upstream repos and are pulled in at build time: + +- `Ontos-AI/knowhere` — Python API + Celery worker (`apps/api`, `apps/worker`, `packages/shared-python`) +- `Ontos-AI/knowhere-dashboard` — Next.js dashboard + +Do not look for or edit app logic here. Changes to API/worker/dashboard behavior must be made upstream. This repo's surface is: `Dockerfile`, `compose.yaml`, `.env.defaults`, `scripts/`, `docs/`, `.github/workflows/`. + +## Verification + +There is **no test, lint, typecheck, or format toolchain** in this repo — do not assume `npm test`, `pytest`, `ruff`, etc. exist. The only verification is the smoke test: + +```bash +./scripts/smoke-test.sh +``` + +It brings up the full stack under `COMPOSE_PROJECT_NAME=knowhere-self-hosted-smoke` on shifted ports (dashboard `13000`, API `15005`, postgres `15432`, redis `16379`, localstack `14566`) and polls `/login` + `/health` for up to 90×2s. + +For shell/Python script edits, sanity-check with `bash -n scripts/*.sh` and `python3 -m py_compile scripts/*.py`. + +## Local image build + +The Dockerfile copies from `.build/sources/{knowhere,knowhere-dashboard}/`, which is gitignored and staged by: + +```bash +./scripts/prepare-sources.sh +``` + +By default it expects sibling checkouts at `../knowhere` and `../knowhere-dashboard` (archive of `HEAD`). Override with `KNOWHERE_API_SOURCE` / `KNOWHERE_API_REF` / `KNOWHERE_DASHBOARD_SOURCE` / `KNOWHERE_DASHBOARD_REF`. Then `docker build .`. + +## Image publishing + +Automated via `.github/workflows/publish-image.yml` only — **manually triggered** (`workflow_dispatch`) with `image_tag`, `api_ref`, `dashboard_ref`, `publish_latest`, `create_github_release`. Builds multi-arch (`linux/amd64,linux/arm64`) and pushes to both GHCR (`ghcr.io/ontos-ai/knowhere`) and Aliyun ACR. Tags matching `-(alpha|beta|rc)` are marked prerelease. Do not push tags by hand to release. + +## Environment configuration + +`compose.yaml` loads `.env.defaults` (committed, **do not put secrets here**) then `.env` (operator overrides, gitignored) via `env_file`. The README tells operators to create a small `.env` with only overrides. + +Runtime defaults are layered further by `scripts/entrypoint.sh` via `setDefault` and several values are **derived** (e.g. `API_DATABASE_URL` from `POSTGRES_*`, `NEXT_PUBLIC_APP_URL` from `DASHBOARD_PUBLIC_URL`, `CELERY_REDIS_URL` from `REDIS_*`). When changing a default, check both `.env.defaults` and `entrypoint.sh` — the entrypoint can override what's in the file. Full variable reference: `docs/configuration.md`. + +Auto-generated secrets (`SECRET_KEY`, `BETTER_AUTH_SECRET`, `USERS_VERIFY_*`, `USERS_RESET_PASSWORD_*`) are persisted in the `knowhere_secrets` volume at `/data/secrets/`; deleting that volume regenerates them. + +## Single-container runtime + +The image runs **three processes** in one container via `scripts/entrypoint.sh` (supervised by `tini`), in this order: + +1. wait for postgres → ensure extensions (`uuid-ossp`, `pg_trgm`) +2. wait for redis +3. create S3 buckets (`scripts/create-storage-buckets.py`) +4. run dashboard drizzle migrations +5. start API (`apps/api/main.py`, port 5005) → wait for `/health` +6. configure S3 event notifications + SNS subscription (`scripts/configure-storage-events.py`) — runs after API so the webhook target is up +7. start worker (`apps/worker/worker.py`) +8. start dashboard (`next start`, port 3000) + +Container exits if any of the three processes exits. Healthcheck hits both `:3000/login` and `:5005/health`. + +Two separate venvs are built in the image: `/opt/knowhere/venvs/api` and `/opt/knowhere/venvs/worker`. Both install from the upstream `uv.lock` with `uv sync --locked --no-dev`. + +## Compose services + +`app` (the combined image), `postgres:15-alpine`, `redis:7-alpine`, `localstack:3.8` (S3/SNS/SQS/IAM/STS with persistence). All host ports bind `127.0.0.1` by default; set `*_HOST_BIND=0.0.0.0` to expose. Named volumes persist data across `docker compose down`. + +LocalStack is reachable at `localhost.localstack.cloud:4566` (network alias) for S3 operations; `S3_ENDPOINT_URL` defaults to that host so presigned URLs work from inside the container. + +## Telemetry + +Anonymous telemetry is **on by default** (`TELEMETRY_ENABLED=true`). Opt out by setting `TELEMETRY_ENABLED=false`. Event schema and privacy bounds are documented in `docs/configuration.md` → "Anonymous product telemetry". + +## Conventions + +- Keep `.env.defaults` and `docs/configuration.md` in sync when adding/removing/renaming env vars — the docs are the operator-facing reference and the file is the executable default. +- `README.md` and `README.zh-CN.md` (plus `docs/configuration.md` / `docs/configuration.zh-CN.md`) are maintained in parallel; update both languages for user-facing changes. +- Branch naming seen in history: `feat/`, `fix/`, `chore/`, `docs/`, `ci/` with `/`. PRs merge into `main`. +- The default image (`KNOWHERE_IMAGE=ghcr.io/ontos-ai/knowhere:latest`) and an Aliyun mirror are both supported; mention both when touching image references. + +## Agent skills + +### Issue tracker + +Local markdown under `.scratch//`. See `docs/agents/issue-tracker.md`. + +### Triage labels + +Canonical defaults (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`) recorded as `Status:` lines in each issue file. See `docs/agents/triage-labels.md`. + +### Domain docs + +Single-context layout (`CONTEXT.md` at repo root, `docs/adr/` for architectural decisions). See `docs/agents/domain.md`. diff --git a/docs/adr/0001-local-mineru-mode.md b/docs/adr/0001-local-mineru-mode.md new file mode 100644 index 0000000..063460a --- /dev/null +++ b/docs/adr/0001-local-mineru-mode.md @@ -0,0 +1,87 @@ +# ADR 0001: Local MinerU mode via forked image patch + +Date: 2026-07-22 + +## Status + +Accepted. Stopgap pending upstream support in the `knowhere` monorepo. + +## Context + +Knowhere's upstream `pdf_service.py` only implements the MinerU **cloud** flow: request an upload URL → upload the PDF → poll batch status → download a ZIP. Self-hosted deployments that want to run fully offline (or against a private MinerU instance on their own network) cannot use the cloud-only batch APIs (`/file-urls/batch`, `/extract/task/batch`, `/extract-results/batch`). Local MinerU exposes a single synchronous `/file_parse` endpoint with a different request schema and a different ZIP layout. + +MinerU is also single-concurrency by default (`max_concurrent_requests=1`), which interacts non-trivially with Knowhere's shard fan-out. + +## Decision + +Ship a **forked Docker image** (`Dockerfile.forked`) that layers a unified diff (`patches/pdf_service.patch`) on top of the pinned upstream image (`ghcr.io/ontos-ai/knowhere:${KNOWHERE_BASE_TAG}`). + +The patch adds: + +1. A `MINERU_LOCAL_MODE` gate at the top of `parse_via_full`, dispatching to a new `parse_via_local`. +2. `parse_via_local` — POSTs the PDF as multipart form data to `{MINERU_URL}/file_parse`. Parameters (`lang_list`, `backend`) are env-overridable via `MINERU_LOCAL_LANG_LIST` and `MINERU_LOCAL_BACKEND`, with `ch`/`pipeline` defaults that match the broadest local MinerU compatibility (Chinese, English, Japanese, Traditional Chinese, Latin; CPU-only). +3. `_flatten_extracted_zip` — flattens the local MinerU ZIP structure (`{stem}/auto/{stem}.md` + `{stem}/auto/images/*`) into `{output_dir}/full.md` + `{output_dir}/images/*`. Hard-fails on zero or multiple `.md` files. +4. A separate local-mode `requests.Session` with `Retry(read=0, ...)` so that a `ReadTimeout` (from queue wait at `max_concurrent_requests=1`) does not cascade into retry storms. 429 responses raise `UnavailableException` using `get_retry_after_seconds` — no quota manager call (local mode has no API key). +5. `MINERU_LOCAL_TIMEOUT` (env-overridable, default 3600s) — per-shard timeout that includes queue wait when `MINERU_SHARD_CONCURRENCY > 1`. + +Distribution is a forked image, not a volume mount, so upgrades are explicit (bump `KNOWHERE_BASE_TAG`) and drift is detectable via `scripts/check-patch-drift.sh` (`patch --dry-run` against the upstream file extracted from the base image). Editing the patch is mediated by `scripts/edit-patch.sh` to avoid hand-maintaining a stale working copy. + +`compose.yaml` adds `extra_hosts: ["host.docker.internal:host-gateway"]` to the app service so that `MINERU_URL=http://host.docker.internal:8000` resolves on Linux Docker Engine (auto-injected on Docker Desktop and OrbStack). + +## Consequences + +### Positive + +- Self-hosted deployments can run without `mineru.net` or any MinerU cloud API key. +- Upgrades are deliberate: `docker compose pull` against `:latest` no longer silently breaks the patched file (it can't — the patch is applied at build time against a pinned base tag). Drift is caught at CI build time by `check-patch-drift.sh`. +- The patch surface is minimal and self-documenting (unified diff). +- Existing cloud-mode deployments are unaffected (`MINERU_LOCAL_MODE=false` by default). + +### Negative + +- Drift maintenance: every upstream `pdf_service.py` change requires re-running `edit-patch.sh` and re-testing. Mitigated by `check-patch-drift.sh` as a CI gate. +- `MINERU_LOCAL_*` env vars are read via `os.environ.get` rather than typed `pydantic_settings` fields, because the image's `shared.core.config.settings` is baked and the forked-image patch cannot edit it. A future upstream PR should add proper typed fields — see "Future work". +- The forked image is published under a personal namespace (`ghcr.io/gdccyuen/knowhere-self-hosted`), not the upstream org. Operators must trust this registry or rebuild locally with `docker compose build`. +- Multi-shard PDFs against a single-concurrency MinerU can pile up in the queue; users must tune `MINERU_SHARD_CONCURRENCY` and `MINERU_LOCAL_TIMEOUT` for very large documents. + +## Pitfalls + +These were encountered during development and are documented here for future maintainers. Most are deployment-time gotchas rather than patch issues. + +### 1. DeepSeek text model used for vision tasks + +`IMAGE_MODEL` and `IMAGE_MODEL_MAX` must be vision-capable. Knowhere sends `image_url` content blocks to these models during PDF image analysis; text-only models like `deepseek-v4-flash` raise `LLMServiceException: unknown variant 'image_url', expected 'text'`. Default the image models to `qwen3.6-flash` (or another vision-capable model) when running local mode. + +### 2. MinerU cloud-only batch APIs called against local MinerU + +Local MinerU's OpenAPI only exposes `/file_parse` (sync), `/tasks` (async submit), `/tasks/{id}` (status), `/tasks/{id}/result` (result). The cloud endpoints (`/file-urls/batch`, `/extract/task/batch`, `/extract-results/batch`) return 404. `MINERU_UPLOAD_MODE_ENABLED=false` does not help — it switches between two cloud modes (URL-based vs upload-based), neither of which works locally. This is the core motivation for the `MINERU_LOCAL_MODE` flag. + +### 3. ZIP from local MinerU has a nested structure + +Local MinerU ZIP extracts to `{stem}/auto/{stem}.md` + `{stem}/auto/images/*.jpg`. Knowhere's downstream code expects `output_dir/full.md` and `images/` at the output dir root. `_flatten_extracted_zip` walks the extracted tree, moves keep-ext files to the root, removes the rest, and renames the first (and only) `.md` to `full.md`. Multiple `.md` files would silently pick the wrong one, so the patch hard-fails in that case. + +### 4. `MINERU_LOCAL_LANG_LIST` does not accept `auto` + +Cloud MinerU accepts `language: "auto"`; local MinerU's `lang_list` enum is `ch, ch_server, korean, ta, te, ka, th, el, arabic, east_slavic, cyrillic, devanagari`. The patch defaults to `ch`, which covers Chinese, English, Japanese, Traditional Chinese, and Latin — broad enough for most use. Setting `auto` against local MinerU returns `400 Language auto not supported`. + +### 5. `return_images=True` is load-bearing + +The default for the `return_images` field is `false`. Without it, the ZIP contains only `og.md` and no `images/` directory — but the markdown references image paths under `images/`. The patch explicitly sends `return_images=True`. + +### 6. `max_concurrent_requests=1` and shard pile-up + +Local MinerU's `/health` reports `max_concurrent_requests=1`. With the default `MINERU_SHARD_CONCURRENCY=3`, three shard HTTP POSTs are issued simultaneously; the second and third queue at MinerU. The session-level `ReadTimeout` clock for the Nth shard starts when **its** POST begins, not when MinerU starts processing it — so a sharded PDF's last shard's effective wall-time budget is `(N-1) * per_shard_parse_time + own_parse_time`. The Q8 decision (separate session with `Retry(read=0)`) prevents a `ReadTimeout` from triggering a urllib3 retry that would push the request to the back of the same queue. + +### 7. `host.docker.internal` resolution on Linux + +Auto-injected on Docker Desktop and OrbStack, but missing on Linux Docker Engine. `compose.yaml` adds `extra_hosts: ["host.docker.internal:host-gateway"]` to the app service (parity with the existing localstack service entry) so that `MINERU_URL=http://host.docker.internal:8000` works on every platform. + +### 8. `MINERU_UPLOAD_MODE_ENABLED` interaction with PPTX cache + +`get_existing_mineru_source_s3_key` is called from `rendered_transform.py` before `parse_pdfs` to reuse a previously-rendered PPTX→PDF artifact. With the default `MINERU_UPLOAD_MODE_ENABLED=true`, S3 URL mode is not active, so cache reuse is skipped and PPTX re-renders on each parse. This is **pre-existing** behavior unrelated to the patch. In local mode, setting `MINERU_UPLOAD_MODE_ENABLED=false` enables PPTX cache reuse via LocalStack S3 presigned URLs. The patch does not force this — operators opt in if they care about PPTX rerun cost. + +## Future work + +- **Upstream contribution.** This patch is a stopgap. A first-class upstream PR to the `knowhere` monorepo would: add `MINERU_LOCAL_MODE`, `MINERU_LOCAL_LANG_LIST`, `MINERU_LOCAL_BACKEND`, `MINERU_LOCAL_TIMEOUT` as typed `pydantic_settings` fields in `shared.core.config.mineru`; refactor `parse_via_full` to dispatch via `settings.MINERU_LOCAL_MODE`; drop the `os.environ.get` reads; and add unit tests for `_flatten_extracted_zip` against representative ZIP fixtures. When a `knowhere` release ≥ X ships this natively, the forked-image approach in this repo should be retired: bump `KNOWHERE_BASE_TAG` to that release, delete `patches/pdf_service.patch`, `Dockerfile.forked`, `scripts/check-patch-drift.sh`, `scripts/edit-patch.sh`, and supersede this ADR. +- **Async `/tasks` API.** If a deployment regularly parses very large PDFs where the sync `/file_parse` timeout becomes the bottleneck, the patch can be extended to use the local MinerU async API (`/tasks` POST → poll `/tasks/{id}` → fetch `/tasks/{id}/result`). This mirrors the cloud batch pattern and avoids holding open HTTP connections during queue wait. Not pursued in this iteration — the sync endpoint is sufficient for typical use. +- **`MINERU_LOCAL_BACKEND=vlm-engine` smoke test.** The current smoke fixture only exercises `pipeline`. A GPU-enabled test environment could verify the VLM backends. diff --git a/docs/agents/domain.md b/docs/agents/domain.md new file mode 100644 index 0000000..0fee144 --- /dev/null +++ b/docs/agents/domain.md @@ -0,0 +1,53 @@ +# Domain Docs + +How the engineering skills should consume this repo's domain documentation when exploring the codebase. + +## Before exploring, read these + +- **`CONTEXT.md`** at the repo root, or +- **`CONTEXT-MAP.md`** at the repo root if it exists — it points at one `CONTEXT.md` per context. Read each one relevant to the topic. +- **`docs/adr/`** — read ADRs that touch the area you're about to work in. In multi-context repos, also check `src//docs/adr/` for context-scoped decisions. + +If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest creating them upfront. The producer skill (`/grill-with-docs`) creates them lazily when terms or decisions actually get resolved. + +## File structure + +Single-context repo (most repos): + +``` +/ +├── CONTEXT.md +├── docs/adr/ +│ ├── 0001-event-sourced-orders.md +│ └── 0002-postgres-for-write-model.md +└── src/ +``` + +Multi-context repo (presence of `CONTEXT-MAP.md` at the root): + +``` +/ +├── CONTEXT-MAP.md +├── docs/adr/ ← system-wide decisions +└── src/ + ├── ordering/ + │ ├── CONTEXT.md + │ └── docs/adr/ ← context-specific decisions + └── billing/ + ├── CONTEXT.md + └── docs/adr/ +``` + +This repo uses the **single-context** layout. + +## Use the glossary's vocabulary + +When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a test name), use the term as defined in `CONTEXT.md`. Don't drift to synonyms the glossary explicitly avoids. + +If the concept you need isn't in the glossary yet, that's a signal — either you're inventing language the project doesn't use (reconsider) or there's a real gap (note it for `/grill-with-docs`). + +## Flag ADR conflicts + +If your output contradicts an existing ADR, surface it explicitly rather than silently overriding: + +> _Contradicts ADR-0007 (event-sourced orders) — but worth reopening because…_ diff --git a/docs/agents/issue-tracker.md b/docs/agents/issue-tracker.md new file mode 100644 index 0000000..a2f08fb --- /dev/null +++ b/docs/agents/issue-tracker.md @@ -0,0 +1,19 @@ +# Issue tracker: Local Markdown + +Issues and PRDs for this repo live as markdown files in `.scratch/`. + +## Conventions + +- One feature per directory: `.scratch//` +- The PRD is `.scratch//PRD.md` +- Implementation issues are `.scratch//issues/-.md`, numbered from `01` +- Triage state is recorded as a `Status:` line near the top of each issue file (see `triage-labels.md` for the role strings) +- Comments and conversation history append to the bottom of the file under a `## Comments` heading + +## When a skill says "publish to the issue tracker" + +Create a new file under `.scratch//` (creating the directory if needed). + +## When a skill says "fetch the relevant ticket" + +Read the file at the referenced path. The user will normally pass the path or the issue number directly. diff --git a/docs/agents/triage-labels.md b/docs/agents/triage-labels.md new file mode 100644 index 0000000..09b9f9c --- /dev/null +++ b/docs/agents/triage-labels.md @@ -0,0 +1,15 @@ +# Triage Labels + +The skills speak in terms of five canonical triage roles. This file maps those roles to the actual label strings used in this repo's issue tracker. + +| Label in mattpocock/skills | Label in our tracker | Meaning | +| -------------------------- | -------------------- | ---------------------------------------- | +| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue | +| `needs-info` | `needs-info` | Waiting on reporter for more information | +| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent | +| `ready-for-human` | `ready-for-human` | Requires human implementation | +| `wontfix` | `wontfix` | Will not be actioned | + +In the local-markdown tracker, these roles appear as a `Status:` line near the top of each issue file, e.g. `Status: ready-for-agent`. + +When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label string from this table. diff --git a/upload-pdf.sh b/upload-pdf.sh new file mode 100755 index 0000000..f80e23a --- /dev/null +++ b/upload-pdf.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Upload a PDF to Knowhere via the API. +# Usage: ./upload-pdf.sh [namespace] [file_path] +# API_KEY - Knowhere API key (from dashboard Settings → API Keys) +# namespace - retrieval namespace (default: "default") +# file_path - path to PDF file (default: "og.pdf") + +API_BASE="http://localhost:5005/api/v1" +API_KEY="${1:?Usage: $0 [namespace] [file_path]}" +NAMESPACE="${2:-default}" +FILE="${3:-og.pdf}" + +[ -f "$FILE" ] || { echo "File not found: $FILE"; exit 1; } + +echo "1. Creating job (namespace=$NAMESPACE, file=$FILE)..." +RESP=$(curl -s -X POST "$API_BASE/jobs" \ + -H "Authorization: Bearer $API_KEY" \ + -H "Content-Type: application/json" \ + -d "{\"source_type\":\"file\",\"file_name\":\"$FILE\",\"namespace\":\"$NAMESPACE\"}") + +JOB_ID=$(echo "$RESP" | python3 -c "import sys,json; print(json.load(sys.stdin)['job_id'])") +UPLOAD_URL=$(echo "$RESP" | python3 -c "import sys,json; print(json.load(sys.stdin)['upload_url'])") +echo " job_id: $JOB_ID" + +echo "2. Uploading file to S3..." +curl -s -X PUT "$UPLOAD_URL" \ + -H "content-type: application/pdf" \ + --data-binary @"$FILE" >/dev/null +echo " uploaded" + +echo "3. Waiting for S3 event propagation..." +sleep 5 + +echo "4. Confirming upload..." +curl -s -X POST "$API_BASE/jobs/$JOB_ID/confirm-upload" \ + -H "Authorization: Bearer $API_KEY" | python3 -m json.tool + +echo "" +echo "Done. Job $JOB_ID is now processing." +echo "Check status: curl -s -H 'Authorization: Bearer $API_KEY' $API_BASE/jobs/$JOB_ID | python3 -m json.tool" From ef5e9f1f8e0492437936722de69ad1254ed98b86 Mon Sep 17 00:00:00 2001 From: Gordon YUEN Date: Fri, 31 Jul 2026 13:57:02 +0800 Subject: [PATCH 2/5] docs: update ADR 0001 with MinerU 3.4.0 findings, add ADR 0002 for raw ZIP audit archive ADR 0001 updated: - Status: upstream PR #233 opened - Pitfalls 9-11: files form field, JSON response (not ZIP), response_format_zip=true behavior - Future work: updated to reflect current state ADR 0002 added: - Decision to permanently archive raw MinerU ZIPs to S3 - Covers both local and cloud mode - Storage: S3 + DB column (not DB blob) - 6-step implementation plan (todos 1/6-6/6) --- docs/adr/0001-local-mineru-mode.md | 35 +++++++- docs/adr/0002-mineru-raw-zip-audit-archive.md | 79 +++++++++++++++++++ 2 files changed, 111 insertions(+), 3 deletions(-) create mode 100644 docs/adr/0002-mineru-raw-zip-audit-archive.md diff --git a/docs/adr/0001-local-mineru-mode.md b/docs/adr/0001-local-mineru-mode.md index 063460a..a8b2a5e 100644 --- a/docs/adr/0001-local-mineru-mode.md +++ b/docs/adr/0001-local-mineru-mode.md @@ -4,7 +4,7 @@ Date: 2026-07-22 ## Status -Accepted. Stopgap pending upstream support in the `knowhere` monorepo. +Accepted. Upstream PR [Ontos-AI/knowhere#233](https://github.com/Ontos-AI/knowhere/pull/233) opened to replace this stopgap with first-class support. Updated with findings from live testing against MinerU 3.4.0 (see pitfalls 9–11). ## Context @@ -80,8 +80,37 @@ Auto-injected on Docker Desktop and OrbStack, but missing on Linux Docker Engine `get_existing_mineru_source_s3_key` is called from `rendered_transform.py` before `parse_pdfs` to reuse a previously-rendered PPTX→PDF artifact. With the default `MINERU_UPLOAD_MODE_ENABLED=true`, S3 URL mode is not active, so cache reuse is skipped and PPTX re-renders on each parse. This is **pre-existing** behavior unrelated to the patch. In local mode, setting `MINERU_UPLOAD_MODE_ENABLED=false` enables PPTX cache reuse via LocalStack S3 presigned URLs. The patch does not force this — operators opt in if they care about PPTX rerun cost. +### 9. Form field is `files`, not `file` (MinerU 3.4.0) + +Local MinerU's `/file_parse` endpoint expects the PDF under the multipart form field name `files` (an array), not `file`. Sending `file` returns `422 Unprocessable Entity: {"detail":[{"type":"missing","loc":["body","files"],"msg":"Field required","input":null}]}`. The upstream PR uses `files = {"files": (filename, file_obj, "application/pdf")}`. + +### 10. Default response is JSON with inline content, not a ZIP (MinerU 3.4.0) + +MinerU 3.4.0's `/file_parse` returns `application/json` by default — not a ZIP as ADR 0001 originally assumed. The JSON response contains: + +```json +{ + "task_id": "...", + "status": "completed", + "result_url": "http://.../tasks/{id}/result", + "results": { + "{stem}": { + "md_content": "# heading ...", + "images": { "hash.jpg": "", ... } + } + } +} +``` + +The `result_url` endpoint also returns JSON (same `results` structure), not a ZIP. The upstream PR (as of commit `4eae0385`) writes `md_content` to `full.md` and base64-decodes images to `images/`. A future revision will switch to `response_format_zip=true` to get a real ZIP (see ADR 0002). + +### 11. `response_format_zip=true` returns a ZIP with nested layout + +When `response_format_zip=true` is set, MinerU 3.4.0 returns `application/zip` with the nested `{stem}/auto/{stem}.md` + `{stem}/auto/images/*` layout that `_flatten_extracted_zip` was originally designed for. Adding `return_original_file=true` also includes `{stem}/auto/{stem}_origin.pdf`. This is the preferred path going forward (see ADR 0002). + ## Future work -- **Upstream contribution.** This patch is a stopgap. A first-class upstream PR to the `knowhere` monorepo would: add `MINERU_LOCAL_MODE`, `MINERU_LOCAL_LANG_LIST`, `MINERU_LOCAL_BACKEND`, `MINERU_LOCAL_TIMEOUT` as typed `pydantic_settings` fields in `shared.core.config.mineru`; refactor `parse_via_full` to dispatch via `settings.MINERU_LOCAL_MODE`; drop the `os.environ.get` reads; and add unit tests for `_flatten_extracted_zip` against representative ZIP fixtures. When a `knowhere` release ≥ X ships this natively, the forked-image approach in this repo should be retired: bump `KNOWHERE_BASE_TAG` to that release, delete `patches/pdf_service.patch`, `Dockerfile.forked`, `scripts/check-patch-drift.sh`, `scripts/edit-patch.sh`, and supersede this ADR. -- **Async `/tasks` API.** If a deployment regularly parses very large PDFs where the sync `/file_parse` timeout becomes the bottleneck, the patch can be extended to use the local MinerU async API (`/tasks` POST → poll `/tasks/{id}` → fetch `/tasks/{id}/result`). This mirrors the cloud batch pattern and avoids holding open HTTP connections during queue wait. Not pursued in this iteration — the sync endpoint is sufficient for typical use. +- **Upstream contribution.** Upstream PR [Ontos-AI/knowhere#233](https://github.com/Ontos-AI/knowhere/pull/233) implements first-class `MINERU_LOCAL_MODE` support: typed `pydantic_settings` fields in `shared.core.config.mineru`, `parse_via_full` dispatch, unit tests for `_flatten_extracted_zip`. When a `knowhere` release ships this natively, bump `KNOWHERE_BASE_TAG` to that release and supersede this ADR. +- **Switch to `response_format_zip=true`.** PR #233 currently handles MinerU 3.4.0's default JSON response (inline `md_content` + base64 images). Switching to `response_format_zip=true` simplifies the code (revert to `_flatten_extracted_zip`, drop base64 decoding) and enables saving the raw ZIP for audit (see ADR 0002). +- **Async `/tasks` API.** If a deployment regularly parses very large PDFs where the sync `/file_parse` timeout becomes the bottleneck, the patch can be extended to use the local MinerU async API (`/tasks` POST → poll `/tasks/{id}` → fetch `/tasks/{id}/result`). Not pursued — sync endpoint is sufficient for typical use. - **`MINERU_LOCAL_BACKEND=vlm-engine` smoke test.** The current smoke fixture only exercises `pipeline`. A GPU-enabled test environment could verify the VLM backends. diff --git a/docs/adr/0002-mineru-raw-zip-audit-archive.md b/docs/adr/0002-mineru-raw-zip-audit-archive.md new file mode 100644 index 0000000..740c399 --- /dev/null +++ b/docs/adr/0002-mineru-raw-zip-audit-archive.md @@ -0,0 +1,79 @@ +# ADR 0002: Save raw MinerU ZIP for permanent audit and re-processing + +Date: 2026-07-31 + +## Status + +Proposed. Implementation tracked as todos 1/6–6/6. Depends on PR [Ontos-AI/knowhere#233](https://github.com/Ontos-AI/knowhere/pull/233). + +## Context + +During live testing of local MinerU mode (ADR 0001), we discovered that MinerU 3.4.0's `/file_parse` can return a ZIP (`response_format_zip=true`) containing the parsed markdown, extracted images, and — with `return_original_file=true` — the original input PDF. This ZIP is the complete raw output of MinerU for a given document. + +Currently, Knowhere discards the raw MinerU output after extracting markdown and images. There is no way to: +- Audit what MinerU actually returned (for debugging parsing quality) +- Re-process a document without re-calling MinerU (which costs API credits in cloud mode) +- Inspect the original PDF alongside MinerU's output + +The requirement is to **permanently archive** the raw MinerU ZIP for every job, in both local and cloud mode. + +## Decision + +Store the raw MinerU ZIP as an S3 artifact, with the S3 key recorded in the `job_results` table. Expose a download endpoint for retrieval. + +### Storage: S3 + DB column (not DB blob) + +The raw ZIP is uploaded to the `knowhere-results` bucket at `results/{job_id}/mineru_raw.zip`. The S3 key is stored in a new nullable `mineru_raw_s3_key TEXT` column on `job_results`. + +This follows the existing pattern — every binary artifact in Knowhere (uploads, result ZIPs, page citation assets) is stored in S3 with paths in the database. Storing blobs directly in PostgreSQL would break the pattern, bloat backups, pressure the connection pool, and complicate replication. + +### Local mode (`parse_via_local`) + +Switch to `response_format_zip=true` + `return_original_file=true`. The response body is a raw ZIP. Upload it to S3 before extracting + flattening. Write the S3 key to a sidecar file (`{output_dir}/_mineru_raw_s3_key.txt`) for the caller to pick up — the parse call chain is 4 layers deep and doesn't return values upward, so a sidecar file is less invasive than threading a return value through. + +### Cloud mode (`poll_mineru_task` → `download_and_extract_zip`) + +`download_and_extract_zip` already saves the ZIP to a temp file (`parsed.zip`), extracts it, then deletes the temp file. Add an optional `on_zip_downloaded: Callable[[Path], None] | None` callback parameter that fires after download but before extract+delete. In `poll_mineru_task`, pass a callback that uploads the ZIP to S3 and writes the S3 key to the same sidecar file. + +### Caller wiring + +After parsing completes, the job result creation code reads `_mineru_raw_s3_key.txt` from `output_dir` and stores the value on `job_results.mineru_raw_s3_key`. + +### Download API + +`GET /api/v2/documents/{document_id}/files/mineru-raw` — follows the existing `page-citation-source` endpoint pattern. Looks up document → current job result → `mineru_raw_s3_key`. Returns 404 if null. Otherwise returns a presigned download URL. + +## Consequences + +### Positive + +- Every MinerU parse is permanently archived with the original input file included. +- Operators can download raw ZIPs for audit without database access. +- Re-processing is possible without re-calling MinerU (useful when iterating on chunking/retrieval config). +- Consistent with existing S3 + DB path pattern. +- `response_format_zip=true` simplifies `parse_via_local` (revert from base64 JSON to ZIP extract+flatten). + +### Negative + +- Additional S3 storage per job (raw ZIP with images + original PDF adds ~1–10 MB per document). +- Sidecar file (`_mineru_raw_s3_key.txt`) is a pragmatic but unconventional IPC mechanism — undocumented knowledge for future maintainers. +- Cloud mode requires modifying the shared `download_and_extract_zip` utility (adding a callback parameter) — minimal but touches shared code. + +## Alternatives considered + +- **DB blob (`bytea` column)**: rejected — breaks the S3+path pattern, bloats backups, pressures connection pool, complicates replication. See discussion in session. +- **Env-gated debug save only**: rejected — requirement is permanent audit, not just development debugging. +- **Thread return value through parse call chain**: rejected — 4 layers deep (`parse_pdfs` → `_parse_pdf_via_shards` → `parse_via_full` → `parse_via_local`/`poll_mineru_task`), too invasive for the value returned. + +## Implementation plan + +| Step | Description | Depends on | +| --- | --- | --- | +| 1/6 | DB migration: add `mineru_raw_s3_key TEXT` to `job_results` | — | +| 2/6 | ORM model: add field to `JobResult` | 1/6 | +| 3/6 | Local mode: switch to ZIP mode, upload to S3, sidecar file | 1/6, 2/6 | +| 4/6 | Cloud mode: add callback to `download_and_extract_zip`, upload to S3 | 1/6, 2/6 | +| 5/6 | Caller wiring: read sidecar, store S3 key on job result | 3/6, 4/6 | +| 6/6 | Download API endpoint + contract test | 1/6, 2/6, 5/6 | + +PRs: 3/6 updates PR #233. 4/6 opens a new PR. 1/6+2/6+5/6+6/6 in a third PR (shared infrastructure). From 6502f618b902543141edba583c3ec2dd3b6d1b8d Mon Sep 17 00:00:00 2001 From: Gordon YUEN Date: Sat, 1 Aug 2026 09:13:08 +0800 Subject: [PATCH 3/5] docs: mark ADR 0002 accepted; document implementation refinements --- docs/adr/0002-mineru-raw-zip-audit-archive.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/adr/0002-mineru-raw-zip-audit-archive.md b/docs/adr/0002-mineru-raw-zip-audit-archive.md index 740c399..64f5935 100644 --- a/docs/adr/0002-mineru-raw-zip-audit-archive.md +++ b/docs/adr/0002-mineru-raw-zip-audit-archive.md @@ -4,7 +4,17 @@ Date: 2026-07-31 ## Status -Proposed. Implementation tracked as todos 1/6–6/6. Depends on PR [Ontos-AI/knowhere#233](https://github.com/Ontos-AI/knowhere/pull/233). +Accepted. Implemented (todos 1/6–6/6 complete) across three upstream PRs: + +- [Ontos-AI/knowhere#233](https://github.com/Ontos-AI/knowhere/pull/233) — step 3/6: local MinerU mode archives the raw ZIP (`response_format_zip=true` + `return_original_file=true`), uploads to `results/{job_id}/mineru_raw.zip`, writes the `_mineru_raw_s3_key.txt` sidecar. +- [Ontos-AI/knowhere#238](https://github.com/Ontos-AI/knowhere/pull/238) — step 4/6: cloud mode archives via `on_zip_downloaded` callback on `download_and_extract_zip`. +- [Ontos-AI/knowhere#239](https://github.com/Ontos-AI/knowhere/pull/239) — steps 1/6, 2/6, 5/6, 6/6: migration + ORM field, caller wiring stores the sidecar value on `job_results.mineru_raw_s3_key`, and `GET /api/v2/documents/{document_id}/files/mineru-raw` returns presigned URL(s). + +Refinements made during implementation (deltas from the plan above): + +- **Sharded jobs** archive one raw ZIP per shard with suffixed keys (`results/{job_id}/mineru_raw_shard0.zip`, ...); the worker merges per-shard sidecars into the main output dir sidecar (newline-joined) before the shard workspace is cleaned up. +- The download endpoint returns a single `url` when one key is stored and a `urls` list for sharded (multi-key) jobs. +- The cloud-mode sidecar is written after polling completes (not inside the callback), because `download_and_extract_zip`'s extraction cleanup would prune a sidecar written during the callback. ## Context From 74d00f6da859ca3a2f979c0667bade132a6a8a1f Mon Sep 17 00:00:00 2001 From: Gordon YUEN Date: Mon, 3 Aug 2026 14:11:32 +0800 Subject: [PATCH 4/5] docs: recommend PDF input over docx; fix upload-pdf.sh arg order MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - README (en/zh): add Input Format Recommendation — convert Office files to PDF before uploading. Documents why (docx pipeline flattens tables, drops footers, no full.md; MinerU office backend unreliable; empirical findings from funding-application testing) and the recommended workflow for cost/savings analysis - upload-pdf.sh: fix usage to [file_path] [namespace] with file_path required, namespace defaulting to 'default' --- README.md | 27 +++++++++++++++++++++++++++ README.zh-CN.md | 21 +++++++++++++++++++++ upload-pdf.sh | 10 +++++----- 3 files changed, 53 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index baac2ba..79f1e72 100644 --- a/README.md +++ b/README.md @@ -102,6 +102,33 @@ Use an official SDK to call the API: - Node.js SDK: [Ontos-AI/knowhere-node-sdk](https://github.com/Ontos-AI/knowhere-node-sdk) - Python SDK: [Ontos-AI/knowhere-python-sdk](https://github.com/Ontos-AI/knowhere-python-sdk) +### Input Format Recommendation + +**Convert Office files to PDF before uploading.** Knowhere parses PDFs with MinerU, which produces +the most complete and structured result (full markdown, table cell structure, footers, and +footnotes). Word documents (`.docx`) are parsed by a separate, simpler pipeline that: + +- flattens tables into plain text, losing the row-to-column value mapping; +- drops page footers and footer-note legends; +- does not emit a `full.md` artifact. + +The MinerU office backend in current releases is not reliable for real-world `.docx` files either. +Empirical testing on government funding-application forms found that submitting `.docx` directly to +MinerU returned an empty markdown (only a page-number header survived), and figures like the "Net +Additional Recurrent Expenditure" row were occasionally lost when PDFs were generated by third-party +converters. The original `.docx` never loses a figure, but without table structure it is hard to +attribute values to columns. + +**Recommended workflow for accurate cost/savings analysis:** + +1. Convert the source `.docx` to PDF with your office suite (e.g. LibreOffice, Microsoft Word, + or Adobe Acrobat) before uploading. +2. Upload the **PDF** to Knowhere. +3. Keep the original `.docx` as a second reference — if a figure looks suspicious, cross-check it + against the source document. Do not rely on a single conversion's output for totals rows. + +This gives you MinerU's full structural output while keeping the original file's fidelity. + ## Common Commands Check service status: diff --git a/README.zh-CN.md b/README.zh-CN.md index 3dbc9e3..e5359cc 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -98,6 +98,27 @@ http://localhost:5005/health - Node.js SDK:[Ontos-AI/knowhere-node-sdk](https://github.com/Ontos-AI/knowhere-node-sdk) - Python SDK:[Ontos-AI/knowhere-python-sdk](https://github.com/Ontos-AI/knowhere-python-sdk) +### 输入格式建议 + +**上传前请先将 Office 文件转换为 PDF。** Knowhere 使用 MinerU 解析 PDF,能够输出最完整、结构化的结果 +(完整 markdown、表格单元格结构、页脚及脚注)。Word 文档(`.docx`)走的是另一套较简单的解析流程,其局限包括: + +- 表格被拍平为纯文本,丢失了「行到列」的数值对应关系; +- 丢弃页脚及页脚注释说明; +- 不产出 `full.md` 产物。 + +当前版本的 MinerU office 后端对真实 `.docx` 文件并不可靠。对政府资助申请表格的实测发现:将 `.docx` +直接提交给 MinerU 返回的是空 markdown(仅保留了页码页眉),而使用第三方转换工具生成的 PDF 偶尔也会丢失 +「净新增经常性开支」等关键数字。原始 `.docx` 从不丢失数字,但缺少表格结构,难以判断数值属于哪一列。 + +**进行准确的成本/节省分析时,推荐流程:** + +1. 上传前先用办公套件(如 LibreOffice、Microsoft Word 或 Adobe Acrobat)将源 `.docx` 转换为 PDF; +2. 将 **PDF** 上传到 Knowhere; +3. 保留原始 `.docx` 作为第二参考——如果某个数字可疑,请对照源文档核对;合计行不要只依赖单次转换的输出。 + +这样既能获得 MinerU 完整的结构化输出,又能保留原始文件的准确性。 + ## 常用命令 查看服务状态: diff --git a/upload-pdf.sh b/upload-pdf.sh index f80e23a..247e216 100755 --- a/upload-pdf.sh +++ b/upload-pdf.sh @@ -2,15 +2,15 @@ set -euo pipefail # Upload a PDF to Knowhere via the API. -# Usage: ./upload-pdf.sh [namespace] [file_path] +# Usage: ./upload-pdf.sh [file_path] [namespace] # API_KEY - Knowhere API key (from dashboard Settings → API Keys) +# file_path - path to PDF file (required) # namespace - retrieval namespace (default: "default") -# file_path - path to PDF file (default: "og.pdf") API_BASE="http://localhost:5005/api/v1" -API_KEY="${1:?Usage: $0 [namespace] [file_path]}" -NAMESPACE="${2:-default}" -FILE="${3:-og.pdf}" +API_KEY="${1:?Usage: $0 [file_path] [namespace]}" +FILE="${2:?Usage: $0 [file_path] [namespace]}" +NAMESPACE="${3:-default}" [ -f "$FILE" ] || { echo "File not found: $FILE"; exit 1; } From a0882b0a10b9ecadad9573178e3d9a8f07f7d1fa Mon Sep 17 00:00:00 2001 From: Gordon YUEN Date: Thu, 6 Aug 2026 23:08:52 +0800 Subject: [PATCH 5/5] docs: document MinerU 3.4.0 office backend failure for docx (pitfall 12) Live testing against real funding-application .docx files showed the office backend returns empty markdown (only a page-number header survives). EMF vector images are skipped by the image loader and the docx converter emits zero paragraphs/tables even without EMFs. Recommend converting Office files to PDF before uploading. --- docs/adr/0001-local-mineru-mode.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/adr/0001-local-mineru-mode.md b/docs/adr/0001-local-mineru-mode.md index a8b2a5e..29c57e6 100644 --- a/docs/adr/0001-local-mineru-mode.md +++ b/docs/adr/0001-local-mineru-mode.md @@ -4,7 +4,7 @@ Date: 2026-07-22 ## Status -Accepted. Upstream PR [Ontos-AI/knowhere#233](https://github.com/Ontos-AI/knowhere/pull/233) opened to replace this stopgap with first-class support. Updated with findings from live testing against MinerU 3.4.0 (see pitfalls 9–11). +Accepted. Upstream PR [Ontos-AI/knowhere#233](https://github.com/Ontos-AI/knowhere/pull/233) opened to replace this stopgap with first-class support. Updated with findings from live testing against MinerU 3.4.0 (see pitfalls 9–12). ## Context @@ -108,6 +108,15 @@ The `result_url` endpoint also returns JSON (same `results` structure), not a ZI When `response_format_zip=true` is set, MinerU 3.4.0 returns `application/zip` with the nested `{stem}/auto/{stem}.md` + `{stem}/auto/images/*` layout that `_flatten_extracted_zip` was originally designed for. Adding `return_original_file=true` also includes `{stem}/auto/{stem}_origin.pdf`. This is the preferred path going forward (see ADR 0002). +### 12. The office backend cannot parse real-world `.docx` files (MinerU 3.4.0) + +Despite advertising DOCX/PPTX/XLSX support, submitting a `.docx` to `/file_parse` (any backend) routes to `mineru.backend.office` which produced an empty markdown in testing: only a page-number header survived (`- 2 -`), and the 682 KB funding-application form yielded zero paragraphs and zero tables. Two contributing causes were observed: + +- **EMF vector images.** The docx embeds `/word/media/*.emf` (form checkbox/border graphics). MinerU's office image loader logs `Skipping EMF image part before Pillow load` and drops them — PIL cannot decode EMF/WMF. +- **Empty model output even without EMFs.** Stripping the EMF parts still produced `middle.json` with empty `para_blocks` and a 0-byte markdown, so the docx converter itself is not yet functional in this build. + +**Recommendation: convert Office files to PDF before uploading to Knowhere.** The PDF pipeline (pipeline/vlm backends) is mature and reliable; the office path is not. Users should use LibreOffice/Word/Acrobat to produce the PDF client-side. Documented in the README ("Input Format Recommendation"). + ## Future work - **Upstream contribution.** Upstream PR [Ontos-AI/knowhere#233](https://github.com/Ontos-AI/knowhere/pull/233) implements first-class `MINERU_LOCAL_MODE` support: typed `pydantic_settings` fields in `shared.core.config.mineru`, `parse_via_full` dispatch, unit tests for `_flatten_extracted_zip`. When a `knowhere` release ships this natively, bump `KNOWHERE_BASE_TAG` to that release and supersede this ADR.