Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,9 @@ UPLOAD_MAX_FILE_BYTES=20971520
UPLOAD_MAX_FILES=10
# Cap of one whole upload request (all files + form overhead); requests without Content-Length are refused.
UPLOAD_MAX_REQUEST_BYTES=41943040
# Uploads per person and hour – a cost cap (every upload starts AI calls). Empty = no cap (local, CI);
# required with APP_ENV=showcase.
UPLOAD_MAX_PER_HOUR=

# --- AI service (services/ai) --------------------------------------------------------------------
# Called by the worker only. The token must equal the service's AI_SERVICE_TOKEN (at least 24 chars).
Expand All @@ -76,6 +79,18 @@ ERP_TIMEOUT_MS=10000
ERP_MOCK_ENABLED=true
ERP_MOCK_FAULTS=

# --- Serverless showcase (Vercel, #59 – docs/technical/deployment-vercel.md) ----------------------
# Without a worker process, `/api/jobs/drain` runs one bounded round of processing + export jobs.
# Bearer secret of that route (at least 24 chars; Vercel Cron sends it). Unset → the route answers 404.
# Generate: `openssl rand -base64 32`. Docker Compose runs the worker and leaves it unset.
# CRON_SECRET=
# "true": also drain once via after() right after upload, approval and reprocess (serverless only).
# With CRON_SECRET or this switch set, AI_SERVICE_TIMEOUT_MS × UPLOAD_MAX_FILES must fit into one
# 300 s function run (e.g. 60000 × 3) – otherwise the configuration is refused.
JOB_DRAIN_INLINE=false
# "true": banner „Demo – nur synthetische Daten" on every page (showcase).
DEMO_MODE=false

# --- Web ---------------------------------------------------------------------------------------
# Host port of the web container.
WEB_PORT=3000
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,11 @@ This file records what changes **in the product** – process and session state
tests against real PostgreSQL + SeaweedFS.
- Request list pages by 50 (keyset on creation time and id, stable while new requests arrive):
"Ältere Anfragen" / "Zurück zum Anfang" keep the filters; an invalid page parameter shows page 1.
- Showcase preparation (Vercel + Supabase): without a worker, jobs run right after upload, approval and
reprocess and via a protected drain route (daily cron); `DEMO_MODE` shows „Demo – nur synthetische
Daten" on every page; runbook `docs/technical/deployment-vercel.md` and a role bootstrap script.
- Upload cap per person and hour (`UPLOAD_MAX_PER_HOUR`, required on the showcase): the upload answers
429 before anything is stored.

### Changed
- AI service logs use the web/worker format: `time` (ISO 8601, `Z`) instead of `ts`, lower-case pino level labels (`warn`, not `WARNING`), `logger` only on library records.
Expand Down
44 changes: 43 additions & 1 deletion docs/decisions/ADR-0001-pilot-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
- **Status:** Accepted – 2026-09-22 by Fluory (orchestrator); every decision D1–D11 was confirmed individually
- **Deviation from the draft:** D8 – the orchestrator chose a Python AI service from day 1 instead of
the drafted recommendation (full TypeScript); the draft recommendation is kept as alternative 1 in D8
- **Amendment 2026-09-24 (D11, also touches D3/D5):** the showcase uses Supabase Postgres + Supabase
Storage instead of Neon + R2 – see "Amendment 2026-09-24" at the end of D11
- **Deciders:** Fluory (orchestrator) · drafted by a Claude session
- **Inputs:** `docs/input/2026-09-22-kundenanfrage.md` (customer request), `PROJECT-START.md` (discovery)
- **Facts verified:** 2026-09-22 against official docs, registries and provider terms (sources at the end).
Expand Down Expand Up @@ -610,6 +612,46 @@ exceptions register (`docs/technical/architecture.md`).
acceptance, including the AI-service hosting spike. *Heuristic.*
**Revisit when** the customer environment is known.

### Amendment 2026-09-24 – Supabase instead of Neon + R2 (decided by Fluory, orchestrator; #59)

**Decision.** The showcase keeps Vercel (Hobby) for the TS app but uses **Supabase** in
`eu-central-1` (Frankfurt) for both stateful parts: **Supabase Postgres** replaces Neon (D3) and
**Supabase Storage through its S3-compatible API** replaces Cloudflare R2 (D5). One vendor less,
one dashboard, the same region for database and files. Local, CI and the production path are
unchanged (PostgreSQL 17 container, SeaweedFS, the customer's choice). Nothing Supabase-specific
enters the code: the app still talks plain PostgreSQL (Drizzle, pg-boss) and the S3 API.

**Consequences.**
- **Connections.** At runtime `app_rw` connects through the **Supavisor transaction pooler (port
6543)** – serverless functions open many short connections. This is pooler-safe: `withTenant()`
sets the tenant with `set_config('app.company_id', …, true)`, i.e. transaction-local; node-postgres
and Drizzle use unnamed prepared statements; pg-boss polls (no LISTEN/NOTIFY, D4) and its maintenance
takes transaction-scoped advisory locks. `statement_timeout` is also set on the role `app_rw`
(bootstrap script), so it holds even if the pooler drops the client's startup parameter. Migrations (`pnpm setup:deploy`, `app_owner`) use the **session
pooler** (port 5432) or the direct connection – DDL and the pg-boss installer need a session.
The smoke check (runbook step 8) reads `SHOW statement_timeout` as `app_rw`.
- **Roles.** `app_owner`/`app_rw` are created once by the operator with `scripts/supabase-bootstrap.sql`
(mirrors `docker/postgres/init/01-roles.sh`; `app_rw` NOBYPASSRLS); the first migration still
refuses unsafe roles. The Supabase `postgres`/`service_role` credentials are never given to the app.
- **Data API.** The schemas `app`, `pgboss` and our auth schema must **not** be added to the exposed
schemas of the Supabase Data API (PostgREST/GraphQL); the app never uses the Data API, and `anon`/
`authenticated` get no grants on our schemas.
- **Schema `auth`.** Supabase reserves the schema `auth` for its own Auth service – our Better Auth
schema of the same name cannot be deployed there. It is renamed to `identity` (#60, PR #63): a fresh
database creates `identity` directly, an existing one renames its own `auth` schema only.
- **Storage.** A **private** bucket; S3 access keys from the Storage settings (server-side only),
endpoint `https://<project-ref>.storage.supabase.co/storage/v1/s3`, path-style URLs. The R2
"EU jurisdiction on the free plan" open point is obsolete.
- **Jobs.** Unchanged from D2: no worker on Vercel. A protected route `/api/jobs/drain`
(`CRON_SECRET`) runs one bounded round of `drain()` + `drainExports()` incl. pg-boss maintenance;
`after()` triggers it after upload, approval and reprocess (`JOB_DRAIN_INLINE=true`). The Hobby cron
runs **at most once per day** – the exceptions-register entry "no unattended retries" still applies.
Worst case of one run (AI timeout × files + windows) must fit into the 300 s function limit;
`loadConfig` enforces it.
- **AI service host** is still open (spike). Recommendation: **Google Cloud Run in the EU** (same GCP
project as Vertex `eu`, container image already exists, no 5 GB package or 300 s limit for docling/OCR).
- **Runbook:** `docs/technical/deployment-vercel.md`.

---

## Summary of the challenged decisions
Expand Down Expand Up @@ -688,7 +730,7 @@ explicit requirements.
- pg-boss maintenance API for serverless `drain()`.
- docling EML/MSG coverage and its provenance granularity for XLSX cells.
- Google Gen AI SDK configuration for the Vertex `eu` endpoint.
- R2 EU jurisdiction on the free plan.
- ~~R2 EU jurisdiction on the free plan.~~ Obsolete – Supabase Storage instead (D11 amendment 2026-09-24).
- The AI-service showcase host (spike).

## Sources (verified 2026-09-22)
Expand Down
3 changes: 2 additions & 1 deletion docs/decisions/INDEX.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# Decision index

ADRs are immutable once accepted; a new ADR supersedes an old one with a note ("supersedes ADR-x").
A dated amendment decided by the orchestrator is appended to the decision it changes and listed here.

| ADR | Date | Title | Status |
|---|---|---|---|
| [ADR-0001](ADR-0001-pilot-architecture.md) | 2026-09-22 | Pilot architecture baseline (D1–D11) | Accepted |
| [ADR-0001](ADR-0001-pilot-architecture.md) | 2026-09-22 | Pilot architecture baseline (D1–D11) | Accepted; amended 2026-09-24 (D11: Supabase instead of Neon + R2) |
17 changes: 17 additions & 0 deletions docs/technical/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,23 @@ Request (NEW), documents, audit event and the `request-process` job commit in on
Duplicates: same `Message-ID` (read from `.eml` only – `.msg` Message-ID extraction is a follow-up)
or the same set of file hashes within the company; checks are serialised per company.

## `GET` / `POST /api/jobs/drain` (shared secret, only with `CRON_SECRET` set)

One bounded drain round for runtimes without a worker (#59): pg-boss maintenance, then processing jobs
(new ones for up to 50 s), then export jobs (up to 20 s) – the job in hand always finishes. Vercel Cron
calls it with GET; operators may use POST. Header `Authorization: Bearer <CRON_SECRET>`, compared in
constant time; no body is read, neither secret nor header is logged. Function limit 300 s.

| Status | Body |
|---|---|
| 200 | `{"processing":{"processed":n,"failed":n,"deadLettered":n},"exports":{"exported":n,"failed":n,"deadLettered":n}}` – counts only |
| 401 | `{"error":{"title":"Nicht autorisiert."}}` + `WWW-Authenticate: Bearer` – missing or wrong secret |
| 404 | `CRON_SECRET` not configured (feature off, e.g. Docker Compose with its worker) |
| 503 | `{"error":{"title":"Verarbeitung derzeit nicht möglich."}}` – invalid configuration or infrastructure error (log: variable names / error class) |

Responses carry `cache-control: no-store`. Concurrent calls, `after()` drains and a worker are safe:
pg-boss hands each job to one caller, and the export stays exactly-once (D9).

## `POST /api/erp-mock/v1/quote-requests` (ERP mock, only with `ERP_MOCK_ENABLED=true`)

The simulated ERP of the pilot (ADR-0001 D9) – contract `contracts/erp-export.openapi.yaml`. Without the
Expand Down
18 changes: 10 additions & 8 deletions docs/technical/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,12 @@ Every new file belongs to one of these modules – otherwise add the module here
| `identity` | `src/features/identity/` | Better Auth, users, companies, roles | public login route | personal (staff) | rate limit, invite-only | built: Better Auth (invite-only, organization + admin plugins), `authorize()`, audited invite, user management (`/users`: roles, deactivate/reactivate, last-admin rule), seed |
| `tenancy` | `src/features/tenancy/` | `withTenant()`, RLS policies | internal | – | forced RLS, `app_rw` without BYPASSRLS | built: `withTenant()`, forced RLS on `app.*`, guard test (every `app` table: `company_id`, forced RLS, only company policies; allow-list empty) |
| `audit` | `src/features/audit/` | append-only audit events | internal | personal (staff) | INSERT/SELECT only | partial: `recordAudit()` (append-only enforced by grants) |
| `jobs` | `src/features/jobs/`, entrypoint `src/worker.ts` | pg-boss, job handlers, `drain()`, worker entrypoint | internal | IDs only | transactional enqueue | built: queues, transactional enqueue, handler, `drain()`, dead letter → ERROR, reprocess, worker loop |
| `jobs` | `src/features/jobs/`, entrypoint `src/worker.ts`, shared wiring `src/job-drain.ts` | pg-boss, job handlers, `drain()`, worker entrypoint, one drain round (processing + export) for worker and serverless | internal | IDs only | transactional enqueue | built: queues, transactional enqueue, handler, `drain()`, dead letter → ERROR, reprocess, worker loop, serverless drain (#59) |
| `storage` | `src/features/storage/` | `BlobStore` port + S3 adapter | internal | confidential | private bucket, access via app routes | built: S3 adapter (put/get/delete, bucket setup, ping) |
| `observability` | `src/features/observability/` | logger, health, request-list ops data | `/api/health` | IDs only | no PII in logs | built: pino JSON logs with a fixed key set (IDs + codes only, tested on a full run), health (database, storage; informational AI-service reachability and queue backlog) |
| `db` | `src/db/`, deploy step `src/setup.ts` | Drizzle schema, migrations, DB roles | internal | – | migrations as owner role | built: roles check, schema `app`, default grants for `app_rw` |
| `config` | `src/config/` | typed runtime configuration, validated at start (zod) | internal | secrets (in memory only) | errors name variables, never values | built |
| `app` | `src/app/` | Next.js routes and pages; composition root `src/app/_server/` (pool, storage client) | `/`, `/login`, `/signup`, `/invite`, `/requests`, `/requests/:id`, `/api/requests`, `/api/documents/:id`, `/api/auth/*`, `/api/health`, `/api/erp-mock/v1/quote-requests` (flag) | – | calls module APIs only (dependency-cruiser) | partial: login, sign-up, invite, home, requests, review page, ERP mock route |
| `config` | `src/config/` | typed runtime configuration, validated at start (zod); serverless drain budget | internal | secrets (in memory only) | errors name variables, never values | built |
| `app` | `src/app/` | Next.js routes and pages; composition root `src/app/_server/` (pool, storage client, serverless drain) | `/`, `/login`, `/signup`, `/invite`, `/requests`, `/requests/:id`, `/api/requests`, `/api/documents/:id`, `/api/auth/*`, `/api/health`, `/api/erp-mock/v1/quote-requests` (flag), `/api/jobs/drain` (`CRON_SECRET`) | – | calls module APIs only (dependency-cruiser); drain route: bearer secret, constant-time | partial: login, sign-up, invite, home, requests, review page, ERP mock route |
| AI service | `services/ai/` | docling parsing, extraction, grounding, evals | internal HTTP | confidential + personal (transient) | bearer token, stateless, no DB/storage access | built: eval runner + 15 weighted synthetic cases + replay gate in CI (#24); `POST /v1/extract` – EML, MSG (recursive attachments), PDF (text layer; scans via OCR with `AI_PDF_OCR=auto`), XLSX (rows), DOCX (paragraphs, table cells) → segments with stable locators, bounded OOXML/MSG parsing, 6 header fields + line items (schema v2, prompt `extract_v2`), normalisers (German numbers, units, dates, calendar weeks → at most `uncertain`), grounding verifier, bearer auth; Vertex adapter with recorded responses (live call unverified) |
| Contracts | `contracts/` | OpenAPI: AI service, ERP export | – | – | contract tests | built: `ai-service.openapi.yaml` (generated from the service), `erp-export.openapi.yaml` (hand-written); TS types + drift tests |

Expand All @@ -49,9 +49,9 @@ Deliberately accepted risks – without an entry here a deviation counts as a de
| Exception | Why accepted | Owner | Expires |
|---|---|---|---|
| No RLS on the `identity` (Better Auth) and `pgboss` schemas | Not company-owned business data; reachable only by server code (ADR-0001 D7) | Fluory | 2026-12-31 (review at M3) |
| Showcase without unattended retries (Vercel Hobby cron once/day) | Showcase only; production runs a worker (D2) | Fluory | when a production-like demo is needed |
| Showcase without unattended retries (Vercel Hobby cron once/day) | Showcase only; production runs a worker (D2). Jobs run via `after()` on upload/approval/reprocess and `/api/jobs/drain` (#59) | Fluory | when a production-like demo is needed |
| Better Auth admin plugin mounted without any holder of its admin role | ADR-0001 D6 names the plugin; decided in #30: kept – its `banned` field implements deactivation (sign-in blocked by the plugin). Nobody holds `platform-admin`, so `/api/auth/admin/*` rejects every caller (tested); user management runs through `identity` | Fluory | 2026-12-31 (review at M3) |
| Upload endpoint without a per-user rate limit | Authenticated staff only; body bounded by `Content-Length` + `UPLOAD_MAX_REQUEST_BYTES` before reading | Fluory | before any public deployment (#19) |
| Upload cap per person only where configured | Local and CI run without `UPLOAD_MAX_PER_HOUR`; the showcase refuses to start without it (#59); concurrent uploads may pass the check together – a cost cap, not an exact quota | Fluory | 2026-12-31 (review at M3) |
| `.msg` uploads checked by OLE signature only | Structure check of Outlook messages needs a CFB parser; files are served only as attachments with `nosniff` and parsed later by the stateless AI service | Fluory | with #23 (MSG parsing) |
| Gemini API free tier for local development | Synthetic data only; never showcase or customer data (D8) | Fluory | when a Vertex development budget exists |

Expand All @@ -63,15 +63,17 @@ worker ─► jobs.drain ─► extraction ─► AI service (bytes in, segments
─► requests(REVIEW) + fields + audit ── one transaction
review ─► corrections + approve ─► requests(APPROVED) + export job + audit
worker ─► export ─► ERP (Idempotency-Key) ─► requests(EXPORTED)
showcase (no worker): after() / cron ─► /api/jobs/drain ─► the same drain round (src/job-drain.ts)
failure at any step ─► retry with backoff ─► dead letter ─► requests(ERROR, visible cause)
```

## External services & interfaces

| Service | Purpose | Environments |
|---|---|---|
| PostgreSQL 17 | all state incl. queue and auth | local container · showcase Neon (aws-eu-central-1) |
| S3-compatible storage | original mails and attachments | local SeaweedFS · showcase Cloudflare R2 (EU jurisdiction) |
| PostgreSQL 17 | all state incl. queue and auth | local container · showcase Supabase Postgres (eu-central-1, Supavisor transaction pooler) |
| S3-compatible storage | original mails and attachments | local SeaweedFS · showcase Supabase Storage (S3 API, private bucket) |
| Vercel (Hobby) | showcase runtime of the TS app, daily cron | showcase only – runbook [deployment-vercel.md](deployment-vercel.md) (ADR-0001 D11 amendment 2026-09-24) |
| Vertex AI (`eu` endpoint, gemini-3.5-flash) | extraction | showcase + customer; local dev may use the Gemini free tier with synthetic data |
| ERP | export target | pilot: `erp-mock`; contract `contracts/erp-export.openapi.yaml` |

Expand All @@ -81,7 +83,7 @@ No secrets in this document; every variable is documented in `.env.example`.

| Role | Created by | Used by | Properties |
|---|---|---|---|
| `app_owner` | `docker/postgres/init/01-roles.sh` (password from env) | migrations (`src/setup.ts`, `MIGRATION_DATABASE_URL`) | owns schema `app`; no superuser, NOBYPASSRLS |
| `app_owner` | `docker/postgres/init/01-roles.sh` (password from env); Supabase: `scripts/supabase-bootstrap.sql` | migrations (`src/setup.ts`, `MIGRATION_DATABASE_URL`) | owns schema `app`; no superuser, NOBYPASSRLS |
| `app_rw` | same | web + worker (`DATABASE_URL`) | USAGE on `app`, no CREATE; DML via default privileges; no superuser, NOBYPASSRLS – RLS always applies |

The first migration refuses to run if either role is missing or could bypass RLS.
Loading
Loading