Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion api/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ Two room sets per group (GitHub #160, ADR 0013): **base `${docType}-${groupId}`*
### Endpoints (`src/endpoints/`)

- `POST /changerequest` (`changeRequest.controller.ts`) — handles JSON and multipart (Fastify `request.parts()`). For multipart, file buffers are matched into `BINARY_REF-{uuid}` placeholders in the JSON payload by `util/patchFileData.ts`. `util/removeDangerousKeys.ts` strips prototype-pollution keys before any further processing. Dispatches by `doc.type` through `changeRequests/documentProcessing/process*Dto.ts`.
- `POST /query` (`query.controller.ts` + `query.service.ts`) — Mango queries. A single universal validator (`validation/query/validateQuery.ts`) enforces top-level shape, a `limit` cap, `use_index` membership in the design-doc registry, an operator policy (no `$regex`/`$where`; `$elemMatch` only on `memberOf`/`availableTranslations`/`parentTags`/`tags`), selector depth/clause caps, and a per-request **language cap for NON-CMS queries** (max distinct `language`-field values — `QUERY_MAX_LANGUAGES`, default 4 = the client's 3-preferred cap + the auto-appended default; CMS queries are exempt since they sync all languages; enforced here, before `query.service` injects the permission-language filter, so it caps the client-requested set) — it does NOT do per-identifier dispatch or restrict selector keys (the old template machinery was removed). `body.identifier` is now only an observability label (expensive-query logs / rate-limit context). `BYPASS_TEMPLATE_VALIDATION=true` is a dev/test escape hatch — never in prod. `QueryService` is the data-leakage boundary: it injects permission filters (`cms ? CmsView : View`, GitHub #160 — `cms:true` without CmsView on a requested group → 403) and (when `cms !== true`) published/expiry filters before `executeFindQuery`, blocks the internal `crypto` doc type, and sets `execution_stats: true` so the controller can log expensive (scan-like) queries and feed the optional per-identity rate limiter (`ratelimit/`, default off via `QUERY_RATE_LIMIT_ENABLED`). For non-`cms` responses it also strips any expired Content doc to a `stripExpiredContent()` cleanup stub before returning (the app's `includeExpired` update-sync path returns expired docs only so the client can prune them — the body must not cross the wire).
- `POST /query` (`query.controller.ts` + `query.service.ts`) — Mango queries. A single universal validator (`validation/query/validateQuery.ts`) enforces top-level shape, a `limit` cap, `use_index` membership in the design-doc registry, an operator policy (no `$regex`/`$where`; `$elemMatch` only on `memberOf`/`availableTranslations`/`parentTags`/`tags`), selector depth/clause caps, and a per-request **language cap for NON-CMS queries** (max distinct `language`-field values — `QUERY_MAX_LANGUAGES`, default 5 = the client's 3-preferred cap + the auto-appended default + 1 language of headroom; CMS queries are exempt since they sync all languages; enforced here, before `query.service` injects the permission-language filter, so it caps the client-requested set) — it does NOT do per-identifier dispatch or restrict selector keys (the old template machinery was removed). `body.identifier` is now only an observability label (expensive-query logs / rate-limit context). `BYPASS_TEMPLATE_VALIDATION=true` is a dev/test escape hatch — never in prod. `QueryService` is the data-leakage boundary: it injects permission filters (`cms ? CmsView : View`, GitHub #160 — `cms:true` without CmsView on a requested group → 403) and (when `cms !== true`) published/expiry filters before `executeFindQuery`, blocks the internal `crypto` doc type, and sets `execution_stats: true` so the controller can log expensive (scan-like) queries and feed the optional per-identity rate limiter (`ratelimit/`, default off via `QUERY_RATE_LIMIT_ENABLED`). For non-`cms` responses it also strips any expired Content doc to a `stripExpiredContent()` cleanup stub before returning (the app's `includeExpired` update-sync path returns expired docs only so the client can prune them — the body must not cross the wire).
- `POST /fts` (`ftsSearch.controller.ts` + `ftsSearch.service.ts`) — server-side full-text search complementing offline FTS (ADR 0010). Reproduces the client's trigram + BM25 ranking against the full corpus using two CouchDB views (`fts-trigram-index` splits each `fts` entry to one row per trigram with doc-level filter metadata in the value; `fts-corpus-stats` serves `_stats` for avg doc length). Permission (`cms ? CmsView : View`, GitHub #160) + (non-`cms`) published/scheduled/expired/language filtering is done in JS from the embedded view-row metadata before the top-K cap, then the top-K docs are fetched (by-key `_all_docs`) for full BM25 + word-match parity (`util/ftsScoring.ts`). (No expired-stripping needed here: the non-`cms` path already drops expired/draft Content, so `/fts` never returns one to a non-CMS caller.) Returns ranked page bodies **trimmed of `fts`/`ftsTokenCount` — display-only, clients must not persist them**. The same endpoint also serves a **strict aux path** for non-Content doctypes (`User`, `Redirect`): when `types` is a single such doctype, `searchAux` reads that doctype's own trigram view (`fts-trigram-index-user` / `fts-trigram-index-redirect`, which emit a named metadata object instead of the Content positional tuple) and does substring-AND on the searchable fields + a field sort + permission/`groups` filtering — no BM25, no corpus stats (`AUX_FTS_CONFIG` in `ftsSearch.service.ts`). The `fts` index for these doctypes is computed in `processUserDto`/`processRedirectDto` (strict-only field configs `USER_FTS_FIELDS`/`REDIRECT_FTS_FIELDS`, no client mirror) and backfilled by schema upgrade v18.
- `GET /storage/storagestatus` (`storageStatus.controller.ts`) — bucket connectivity probe; requires `View` on the `Storage` doc.

Expand Down
7 changes: 4 additions & 3 deletions api/src/configuration.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,9 @@ export type QueryConfig = {
/**
* Maximum distinct languages a NON-CMS query may reference (via `language` field constraints).
* Requests above this are rejected with 400. Guards query cost; CMS queries are exempt (they
* sync all languages). Keep in step with the client's preferred-language cap (cap + 1 for the
* auto-appended default). Environment variable: QUERY_MAX_LANGUAGES (default 4).
* sync all languages). The client references at most its preferred cap (3) + 1 auto-appended
* default = 4; the default here keeps one language of headroom (5) so the boundary isn't exact.
* Environment variable: QUERY_MAX_LANGUAGES (default 5).
*/
maxLanguages: number;
/**
Expand Down Expand Up @@ -118,7 +119,7 @@ export default () =>
} as SyncConfig,
query: {
maxLimit: parseInt(process.env.QUERY_MAX_LIMIT, 10) || 500,
maxLanguages: parseInt(process.env.QUERY_MAX_LANGUAGES, 10) || 4,
maxLanguages: parseInt(process.env.QUERY_MAX_LANGUAGES, 10) || 5,
expensiveDocsExamined: parseInt(process.env.QUERY_EXPENSIVE_DOCS_EXAMINED, 10) || 1000,
expensiveExaminedRatio: parseInt(process.env.QUERY_EXPENSIVE_EXAMINED_RATIO, 10) || 10,
rateLimit: {
Expand Down
5 changes: 3 additions & 2 deletions api/src/endpoints/query.controller.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import {
} from "@nestjs/common";
import { QueryService } from "./query.service";
import { MongoQueryDto } from "../dto/MongoQueryDto";
import { validateQuery } from "../validation/query/validateQuery";
import { validateQuery, DEFAULT_MAX_LANGUAGES } from "../validation/query/validateQuery";
import { ConfigService } from "@nestjs/config";
import { WINSTON_MODULE_PROVIDER } from "nest-winston";
import { Logger } from "winston";
Expand Down Expand Up @@ -62,7 +62,8 @@ export class QueryController {
this.configService.get<boolean>("validation.bypassTemplateValidation") || false;

const maxLimit = this.configService.get<number>("query.maxLimit") ?? 500;
const maxLanguages = this.configService.get<number>("query.maxLanguages") ?? 4;
const maxLanguages =
this.configService.get<number>("query.maxLanguages") ?? DEFAULT_MAX_LANGUAGES;

let validationResult = { valid: true, error: "" };
if (!bypassValidation) validationResult = validateQuery(body, { maxLimit, maxLanguages });
Expand Down
10 changes: 6 additions & 4 deletions api/src/validation/query/validateQuery.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,13 @@ export const DEFAULT_MAX_LIMIT = 500;

/**
* Fallback language cap when the caller doesn't supply one. The app lets a user pick at most 3
* preferred languages, with the default (English) auto-appended for display — so a legitimate
* non-CMS content query references at most 4 distinct languages (sync keep ≤3; display ≤4). Keep
* this in step with the client's preferred-language cap (cap + 1 for the auto-appended default).
* preferred languages (`MAX_PREFERRED_LANGUAGES` in app/), with the default (English) auto-appended
* for display — so a legitimate non-CMS content query references at most 3 + 1 = 4 distinct
* languages. We keep one language of HEADROOM (→ 5) so the boundary isn't exact: a transient
* over-cap set, or raising the client's preferred cap by one, no longer instantly gets queries
* rejected. (Raising the client cap by more than one still needs this bumped to match.)
*/
export const DEFAULT_MAX_LANGUAGES = 4;
export const DEFAULT_MAX_LANGUAGES = 5;

/**
* DoS guards on the selector shape. A pathological selector (deeply nested logical
Expand Down
4 changes: 3 additions & 1 deletion app/src/globalConfig.ts
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,9 @@ watch(
* Product cap on how many languages a user may prefer / sync. Enforced authoritatively by the API
* (`QUERY_MAX_LANGUAGES`); these client-side caps keep the UI within it and give good UX. The
* display default (English) is auto-appended in `appDisplayLanguageIdsAsRef` and is NOT counted
* against the preferred cap, so a content query references at most cap + 1 languages.
* against the preferred cap, so a content query references at most cap + 1 languages. The API cap
* keeps one language of headroom above that, so bumping this by one won't need a lock-step API
* change (bumping by more than one does).
*/
export const MAX_PREFERRED_LANGUAGES = 3;
export const MAX_SYNCED_LANGUAGES = 3;
Expand Down
Loading