diff --git a/api/CLAUDE.md b/api/CLAUDE.md index 2765a27ed0..97f52077ff 100644 --- a/api/CLAUDE.md +++ b/api/CLAUDE.md @@ -84,7 +84,7 @@ Two room sets per group (GitHub #160, ADR 0013): **base `${docType}-${groupId}`* ### Endpoints (`src/endpoints/`) - `POST /changerequest` (`changeRequest.controller.ts`) — handles JSON and multipart (Fastify `request.parts()`). For multipart, file buffers are matched into `BINARY_REF-{uuid}` placeholders in the JSON payload by `util/patchFileData.ts`. `util/removeDangerousKeys.ts` strips prototype-pollution keys before any further processing. Dispatches by `doc.type` through `changeRequests/documentProcessing/process*Dto.ts`. -- `POST /query` (`query.controller.ts` + `query.service.ts`) — Mango queries. A single universal validator (`validation/query/validateQuery.ts`) enforces top-level shape, a `limit` cap, `use_index` membership in the design-doc registry, an operator policy (no `$regex`/`$where`; `$elemMatch` only on `memberOf`/`availableTranslations`/`parentTags`/`tags`), selector depth/clause caps, and a per-request **language cap for NON-CMS queries** (max distinct `language`-field values — `QUERY_MAX_LANGUAGES`, default 4 = the client's 3-preferred cap + the auto-appended default; CMS queries are exempt since they sync all languages; enforced here, before `query.service` injects the permission-language filter, so it caps the client-requested set) — it does NOT do per-identifier dispatch or restrict selector keys (the old template machinery was removed). `body.identifier` is now only an observability label (expensive-query logs / rate-limit context). `BYPASS_TEMPLATE_VALIDATION=true` is a dev/test escape hatch — never in prod. `QueryService` is the data-leakage boundary: it injects permission filters (`cms ? CmsView : View`, GitHub #160 — `cms:true` without CmsView on a requested group → 403) and (when `cms !== true`) published/expiry filters before `executeFindQuery`, blocks the internal `crypto` doc type, and sets `execution_stats: true` so the controller can log expensive (scan-like) queries and feed the optional per-identity rate limiter (`ratelimit/`, default off via `QUERY_RATE_LIMIT_ENABLED`). For non-`cms` responses it also strips any expired Content doc to a `stripExpiredContent()` cleanup stub before returning (the app's `includeExpired` update-sync path returns expired docs only so the client can prune them — the body must not cross the wire). +- `POST /query` (`query.controller.ts` + `query.service.ts`) — Mango queries. A single universal validator (`validation/query/validateQuery.ts`) enforces top-level shape, a `limit` cap, `use_index` membership in the design-doc registry, an operator policy (no `$regex`/`$where`; `$elemMatch` only on `memberOf`/`availableTranslations`/`parentTags`/`tags`), selector depth/clause caps, and a per-request **language cap for NON-CMS queries** (max distinct `language`-field values — `QUERY_MAX_LANGUAGES`, default 5 = the client's 3-preferred cap + the auto-appended default + 1 language of headroom; CMS queries are exempt since they sync all languages; enforced here, before `query.service` injects the permission-language filter, so it caps the client-requested set) — it does NOT do per-identifier dispatch or restrict selector keys (the old template machinery was removed). `body.identifier` is now only an observability label (expensive-query logs / rate-limit context). `BYPASS_TEMPLATE_VALIDATION=true` is a dev/test escape hatch — never in prod. `QueryService` is the data-leakage boundary: it injects permission filters (`cms ? CmsView : View`, GitHub #160 — `cms:true` without CmsView on a requested group → 403) and (when `cms !== true`) published/expiry filters before `executeFindQuery`, blocks the internal `crypto` doc type, and sets `execution_stats: true` so the controller can log expensive (scan-like) queries and feed the optional per-identity rate limiter (`ratelimit/`, default off via `QUERY_RATE_LIMIT_ENABLED`). For non-`cms` responses it also strips any expired Content doc to a `stripExpiredContent()` cleanup stub before returning (the app's `includeExpired` update-sync path returns expired docs only so the client can prune them — the body must not cross the wire). - `POST /fts` (`ftsSearch.controller.ts` + `ftsSearch.service.ts`) — server-side full-text search complementing offline FTS (ADR 0010). Reproduces the client's trigram + BM25 ranking against the full corpus using two CouchDB views (`fts-trigram-index` splits each `fts` entry to one row per trigram with doc-level filter metadata in the value; `fts-corpus-stats` serves `_stats` for avg doc length). Permission (`cms ? CmsView : View`, GitHub #160) + (non-`cms`) published/scheduled/expired/language filtering is done in JS from the embedded view-row metadata before the top-K cap, then the top-K docs are fetched (by-key `_all_docs`) for full BM25 + word-match parity (`util/ftsScoring.ts`). (No expired-stripping needed here: the non-`cms` path already drops expired/draft Content, so `/fts` never returns one to a non-CMS caller.) Returns ranked page bodies **trimmed of `fts`/`ftsTokenCount` — display-only, clients must not persist them**. The same endpoint also serves a **strict aux path** for non-Content doctypes (`User`, `Redirect`): when `types` is a single such doctype, `searchAux` reads that doctype's own trigram view (`fts-trigram-index-user` / `fts-trigram-index-redirect`, which emit a named metadata object instead of the Content positional tuple) and does substring-AND on the searchable fields + a field sort + permission/`groups` filtering — no BM25, no corpus stats (`AUX_FTS_CONFIG` in `ftsSearch.service.ts`). The `fts` index for these doctypes is computed in `processUserDto`/`processRedirectDto` (strict-only field configs `USER_FTS_FIELDS`/`REDIRECT_FTS_FIELDS`, no client mirror) and backfilled by schema upgrade v18. - `GET /storage/storagestatus` (`storageStatus.controller.ts`) — bucket connectivity probe; requires `View` on the `Storage` doc. diff --git a/api/src/configuration.ts b/api/src/configuration.ts index 6e7cc3ed53..11af6531c2 100644 --- a/api/src/configuration.ts +++ b/api/src/configuration.ts @@ -36,8 +36,9 @@ export type QueryConfig = { /** * Maximum distinct languages a NON-CMS query may reference (via `language` field constraints). * Requests above this are rejected with 400. Guards query cost; CMS queries are exempt (they - * sync all languages). Keep in step with the client's preferred-language cap (cap + 1 for the - * auto-appended default). Environment variable: QUERY_MAX_LANGUAGES (default 4). + * sync all languages). The client references at most its preferred cap (3) + 1 auto-appended + * default = 4; the default here keeps one language of headroom (5) so the boundary isn't exact. + * Environment variable: QUERY_MAX_LANGUAGES (default 5). */ maxLanguages: number; /** @@ -118,7 +119,7 @@ export default () => } as SyncConfig, query: { maxLimit: parseInt(process.env.QUERY_MAX_LIMIT, 10) || 500, - maxLanguages: parseInt(process.env.QUERY_MAX_LANGUAGES, 10) || 4, + maxLanguages: parseInt(process.env.QUERY_MAX_LANGUAGES, 10) || 5, expensiveDocsExamined: parseInt(process.env.QUERY_EXPENSIVE_DOCS_EXAMINED, 10) || 1000, expensiveExaminedRatio: parseInt(process.env.QUERY_EXPENSIVE_EXAMINED_RATIO, 10) || 10, rateLimit: { diff --git a/api/src/endpoints/query.controller.ts b/api/src/endpoints/query.controller.ts index bc36f62e72..0e5b2e8a74 100644 --- a/api/src/endpoints/query.controller.ts +++ b/api/src/endpoints/query.controller.ts @@ -13,7 +13,7 @@ import { } from "@nestjs/common"; import { QueryService } from "./query.service"; import { MongoQueryDto } from "../dto/MongoQueryDto"; -import { validateQuery } from "../validation/query/validateQuery"; +import { validateQuery, DEFAULT_MAX_LANGUAGES } from "../validation/query/validateQuery"; import { ConfigService } from "@nestjs/config"; import { WINSTON_MODULE_PROVIDER } from "nest-winston"; import { Logger } from "winston"; @@ -62,7 +62,8 @@ export class QueryController { this.configService.get("validation.bypassTemplateValidation") || false; const maxLimit = this.configService.get("query.maxLimit") ?? 500; - const maxLanguages = this.configService.get("query.maxLanguages") ?? 4; + const maxLanguages = + this.configService.get("query.maxLanguages") ?? DEFAULT_MAX_LANGUAGES; let validationResult = { valid: true, error: "" }; if (!bypassValidation) validationResult = validateQuery(body, { maxLimit, maxLanguages }); diff --git a/api/src/validation/query/validateQuery.ts b/api/src/validation/query/validateQuery.ts index b601744eaf..732df6811c 100644 --- a/api/src/validation/query/validateQuery.ts +++ b/api/src/validation/query/validateQuery.ts @@ -23,11 +23,13 @@ export const DEFAULT_MAX_LIMIT = 500; /** * Fallback language cap when the caller doesn't supply one. The app lets a user pick at most 3 - * preferred languages, with the default (English) auto-appended for display — so a legitimate - * non-CMS content query references at most 4 distinct languages (sync keep ≤3; display ≤4). Keep - * this in step with the client's preferred-language cap (cap + 1 for the auto-appended default). + * preferred languages (`MAX_PREFERRED_LANGUAGES` in app/), with the default (English) auto-appended + * for display — so a legitimate non-CMS content query references at most 3 + 1 = 4 distinct + * languages. We keep one language of HEADROOM (→ 5) so the boundary isn't exact: a transient + * over-cap set, or raising the client's preferred cap by one, no longer instantly gets queries + * rejected. (Raising the client cap by more than one still needs this bumped to match.) */ -export const DEFAULT_MAX_LANGUAGES = 4; +export const DEFAULT_MAX_LANGUAGES = 5; /** * DoS guards on the selector shape. A pathological selector (deeply nested logical diff --git a/app/src/globalConfig.ts b/app/src/globalConfig.ts index d409373dc4..c57aee9aa3 100644 --- a/app/src/globalConfig.ts +++ b/app/src/globalConfig.ts @@ -130,7 +130,9 @@ watch( * Product cap on how many languages a user may prefer / sync. Enforced authoritatively by the API * (`QUERY_MAX_LANGUAGES`); these client-side caps keep the UI within it and give good UX. The * display default (English) is auto-appended in `appDisplayLanguageIdsAsRef` and is NOT counted - * against the preferred cap, so a content query references at most cap + 1 languages. + * against the preferred cap, so a content query references at most cap + 1 languages. The API cap + * keeps one language of headroom above that, so bumping this by one won't need a lock-step API + * change (bumping by more than one does). */ export const MAX_PREFERRED_LANGUAGES = 3; export const MAX_SYNCED_LANGUAGES = 3;