diff --git a/.env.example b/.env.example index 8b03541f5..fd81082cd 100644 --- a/.env.example +++ b/.env.example @@ -21,6 +21,36 @@ FIRST_TREE_DATABASE_URL=postgresql://firsttree:firsttree@localhost:5432/firsttre # Bind address — MUST be 0.0.0.0 in Docker; 127.0.0.1 for local FIRST_TREE_HOST=0.0.0.0 +# S3-compatible object storage for attachment/avatar payloads (any of AWS +# S3, Cloudflare R2, MinIO). Local dev: docker compose up -d provides MinIO +# at localhost:9000 with these exact values. Without this group the server +# boots but rejects attachment uploads with 503 until it is configured; +# after configuring, move pre-existing payloads out of PostgreSQL with: +# pnpm --filter @first-tree/server migrate:attachments +FIRST_TREE_S3_BUCKET=firsttree-attachments +FIRST_TREE_S3_ENDPOINT=http://localhost:9000 +FIRST_TREE_S3_ACCESS_KEY_ID=firsttree +FIRST_TREE_S3_SECRET_ACCESS_KEY=firsttree-minio +# Path-style addressing is required for MinIO; leave false for AWS S3/R2. +FIRST_TREE_S3_FORCE_PATH_STYLE=true +# FIRST_TREE_S3_REGION=us-east-1 +# Browser-reachable endpoint for redirect-mode presigned URLs when the +# server reaches storage over an internal address. +# FIRST_TREE_S3_PUBLIC_ENDPOINT= + +# Attachment governance (defaults shown). Downloads: proxy streams bytes +# through the server and works everywhere; redirect answers 302 with a +# <=5 min presigned URL and requires a browser-reachable bucket with CORS +# for the web origin. Quotas are hard rejects (413/422); the governed "2 GB" +# byte quota is implemented as 2 GiB (2^31). +# FIRST_TREE_ATTACHMENT_DOWNLOAD_MODE=proxy +# FIRST_TREE_ATTACHMENT_ORG_QUOTA_BYTES=2147483648 +# FIRST_TREE_ATTACHMENT_ORG_QUOTA_COUNT=1000 +# FIRST_TREE_ATTACHMENT_SWEEP_INTERVAL_SECONDS=900 +# FIRST_TREE_ATTACHMENT_ORPHAN_GRACE_SECONDS=86400 +# FIRST_TREE_ATTACHMENT_PENDING_TTL_SECONDS=3600 +# FIRST_TREE_ATTACHMENT_MAX_CONCURRENT_UPLOADS_PER_UPLOADER=4 + # ┌───────────────────────────────────────────────────────────────────────────┐ # │ Server (SaaS internal) — Optional │ # └───────────────────────────────────────────────────────────────────────────┘ diff --git a/AGENTS.md b/AGENTS.md index 36d052ef8..ce650d8d3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -66,7 +66,7 @@ Operator-only flows such as `login`, `daemon install`, and `agent create` belong ## Architecture Rules - **Package boundaries:** Server, Client, Command, and Web are independently packaged/deployed and share code through `@first-tree/shared`. The CLI is the user-facing command surface and depends only on Client + Shared. Server ships separately as the SaaS Docker image. -- **Server state:** Server is stateless. PostgreSQL is the only persistence/queue/notification backend; do not add Redis or MQ. +- **Server state:** Server is stateless. PostgreSQL is the only relational/queue/notification backend; do not add Redis or MQ. Binary attachment and avatar payloads live in S3-compatible object storage (`FIRST_TREE_S3_*`); PostgreSQL keeps their metadata only. - **Unified user-JWT auth:** A single user JWT authorizes Web/Admin API calls and every agent the user manages on the client WebSocket. Route classification, JWT shape, and scope helpers live in [docs/development/http-path-conventions.md](docs/development/http-path-conventions.md). Channel homes live in [docs/development/local-dev-isolation.md](docs/development/local-dev-isolation.md). Agents bind via `agents.client_id` + `agent:pinned`; R-RUN is re-evaluated at every `agent:bind`. Switching users goes through `first-tree login ` and the local-client switch path; `logout --purge` retires the current server client and cuts its runtime routes before destructive local cleanup, after which cleared agents can be moved to a new connected runtime from Web. - **Inbox boundary:** Server writes to Inbox; Client pulls / receives WebSocket notifications. Delivery is at-least-once; Client deduplicates. - **Agent identity:** Agents are managed by the server Admin API. Agent profile markdown lives in `agents.profile`. Context Tree integration is optional and injected by Client at workspace startup. diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 0f92946ff..2c39ab9b0 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -30,6 +30,18 @@ Run the server and web commands in separate terminals. The server command loads the root `.env` file and enables the local dev GitHub callback stub via its package script. +`docker compose up -d` starts PostgreSQL (relational data) and MinIO +(S3-compatible object storage for attachment/avatar payloads; console at +`http://localhost:9001`). The `.env.example` values point the server at the +compose MinIO out of the box, and the server creates its bucket on boot. +Without the `FIRST_TREE_S3_*` group the server still runs, but attachment +uploads answer 503 until storage is configured. Deployments upgrading from +a pre-object-storage version move existing payloads out of PostgreSQL with +`pnpm --filter @first-tree/server migrate:attachments` (idempotent, +live-safe; see `.env.example`). Server tests need neither service +pre-started — vitest provisions its own containers (or uses the +`CI_DATABASE_URL` / `CI_S3_ENDPOINT` escape hatches). + ## Local URLs - API server: `http://127.0.0.1:8000` @@ -53,6 +65,11 @@ FIRST_TREE_DATABASE_URL=postgresql://firsttree:firsttree@localhost:5432/firsttre FIRST_TREE_HOST=127.0.0.1 FIRST_TREE_PORT=8000 FIRST_TREE_CHANNEL=dev +FIRST_TREE_S3_BUCKET=firsttree-attachments +FIRST_TREE_S3_ENDPOINT=http://localhost:9000 +FIRST_TREE_S3_ACCESS_KEY_ID=firsttree +FIRST_TREE_S3_SECRET_ACCESS_KEY=firsttree-minio +FIRST_TREE_S3_FORCE_PATH_STYLE=true ``` The local `dev` channel can auto-generate `FIRST_TREE_JWT_SECRET` and diff --git a/docker-compose.yml b/docker-compose.yml index 4cf8ac0bb..4f9faeeef 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -15,5 +15,25 @@ services: timeout: 3s retries: 5 + # S3-compatible object storage for attachment/avatar payloads. The server + # auto-creates the bucket on boot; console at http://localhost:9001. + minio: + image: minio/minio:RELEASE.2025-04-22T22-12-26Z + command: server /data --console-address ":9001" + ports: + - "${MINIO_PORT:-9000}:9000" + - "${MINIO_CONSOLE_PORT:-9001}:9001" + environment: + MINIO_ROOT_USER: ${MINIO_ROOT_USER:-firsttree} + MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-firsttree-minio} + volumes: + - miniodata:/data + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:9000/minio/health/ready || exit 1"] + interval: 5s + timeout: 3s + retries: 5 + volumes: pgdata: + miniodata: diff --git a/packages/server/drizzle/0083_overconfident_maelstrom.sql b/packages/server/drizzle/0083_overconfident_maelstrom.sql new file mode 100644 index 000000000..0fcb52034 --- /dev/null +++ b/packages/server/drizzle/0083_overconfident_maelstrom.sql @@ -0,0 +1,18 @@ +CREATE TABLE "attachment_references" ( + "attachment_id" text NOT NULL, + "message_id" text NOT NULL, + "created_at" timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT "attachment_references_attachment_id_message_id_pk" PRIMARY KEY("attachment_id","message_id") +); +--> statement-breakpoint +ALTER TABLE "attachments" ALTER COLUMN "data" DROP NOT NULL;--> statement-breakpoint +ALTER TABLE "agents" ADD COLUMN "avatar_object_key" text;--> statement-breakpoint +ALTER TABLE "attachments" ADD COLUMN "organization_id" text;--> statement-breakpoint +ALTER TABLE "attachments" ADD COLUMN "object_key" text;--> statement-breakpoint +ALTER TABLE "attachments" ADD COLUMN "state" text DEFAULT 'stored' NOT NULL;--> statement-breakpoint +ALTER TABLE "attachment_references" ADD CONSTRAINT "attachment_references_attachment_id_attachments_id_fk" FOREIGN KEY ("attachment_id") REFERENCES "public"."attachments"("id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint +ALTER TABLE "attachment_references" ADD CONSTRAINT "attachment_references_message_id_messages_id_fk" FOREIGN KEY ("message_id") REFERENCES "public"."messages"("id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint +CREATE INDEX "attachment_references_message_id_idx" ON "attachment_references" USING btree ("message_id");--> statement-breakpoint +ALTER TABLE "attachments" ADD CONSTRAINT "attachments_organization_id_organizations_id_fk" FOREIGN KEY ("organization_id") REFERENCES "public"."organizations"("id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint +CREATE INDEX "attachments_org_state_idx" ON "attachments" USING btree ("organization_id","state");--> statement-breakpoint +CREATE INDEX "attachments_state_created_at_idx" ON "attachments" USING btree ("state","created_at"); \ No newline at end of file diff --git a/packages/server/drizzle/LATEST b/packages/server/drizzle/LATEST index 31890704a..24d7d3e74 100644 --- a/packages/server/drizzle/LATEST +++ b/packages/server/drizzle/LATEST @@ -1 +1 @@ -0082_heavy_moonstone +0083_overconfident_maelstrom diff --git a/packages/server/drizzle/meta/0083_snapshot.json b/packages/server/drizzle/meta/0083_snapshot.json new file mode 100644 index 000000000..d041e1f97 --- /dev/null +++ b/packages/server/drizzle/meta/0083_snapshot.json @@ -0,0 +1,6008 @@ +{ + "id": "594a9194-0273-4433-a1a9-9c2d8f30364c", + "prevId": "65938a6b-a650-462c-a831-0433e6d756e3", + "version": "7", + "dialect": "postgresql", + "tables": { + "public.agent_chat_sessions": { + "name": "agent_chat_sessions", + "schema": "", + "columns": { + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "chat_id": { + "name": "chat_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "state": { + "name": "state", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "runtime_state": { + "name": "runtime_state", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'idle'" + }, + "runtime_state_at": { + "name": "runtime_state_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_agent_chat_sessions_chat_agent": { + "name": "idx_agent_chat_sessions_chat_agent", + "columns": [ + { + "expression": "chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "agent_chat_sessions_agent_id_agents_uuid_fk": { + "name": "agent_chat_sessions_agent_id_agents_uuid_fk", + "tableFrom": "agent_chat_sessions", + "tableTo": "agents", + "columnsFrom": [ + "agent_id" + ], + "columnsTo": [ + "uuid" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "agent_chat_sessions_chat_id_chats_id_fk": { + "name": "agent_chat_sessions_chat_id_chats_id_fk", + "tableFrom": "agent_chat_sessions", + "tableTo": "chats", + "columnsFrom": [ + "chat_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "agent_chat_sessions_agent_id_chat_id_pk": { + "name": "agent_chat_sessions_agent_id_chat_id_pk", + "columns": [ + "agent_id", + "chat_id" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.agent_configs": { + "name": "agent_configs", + "schema": "", + "columns": { + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "version": { + "name": "version", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 1 + }, + "payload": { + "name": "payload", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.agent_presence": { + "name": "agent_presence", + "schema": "", + "columns": { + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'offline'" + }, + "instance_id": { + "name": "instance_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "connected_at": { + "name": "connected_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "client_id": { + "name": "client_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "runtime_type": { + "name": "runtime_type", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "runtime_version": { + "name": "runtime_version", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "runtime_state": { + "name": "runtime_state", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "active_sessions": { + "name": "active_sessions", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "total_sessions": { + "name": "total_sessions", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "runtime_updated_at": { + "name": "runtime_updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + } + }, + "indexes": {}, + "foreignKeys": { + "agent_presence_agent_id_agents_uuid_fk": { + "name": "agent_presence_agent_id_agents_uuid_fk", + "tableFrom": "agent_presence", + "tableTo": "agents", + "columnsFrom": [ + "agent_id" + ], + "columnsTo": [ + "uuid" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "agent_presence_client_id_clients_id_fk": { + "name": "agent_presence_client_id_clients_id_fk", + "tableFrom": "agent_presence", + "tableTo": "clients", + "columnsFrom": [ + "client_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.agent_resource_bindings": { + "name": "agent_resource_bindings", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "mode": { + "name": "mode", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "resource_id": { + "name": "resource_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "replaces_resource_id": { + "name": "replaces_resource_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "inline_prompt_body": { + "name": "inline_prompt_body", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "repo_ref": { + "name": "repo_ref", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "repo_local_path": { + "name": "repo_local_path", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "order": { + "name": "order", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_agent_resource_bindings_agent": { + "name": "idx_agent_resource_bindings_agent", + "columns": [ + { + "expression": "agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "type", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_agent_resource_bindings_resource": { + "name": "idx_agent_resource_bindings_resource", + "columns": [ + { + "expression": "resource_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_agent_resource_bindings_replaces": { + "name": "idx_agent_resource_bindings_replaces", + "columns": [ + { + "expression": "replaces_resource_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "agent_resource_bindings_organization_id_organizations_id_fk": { + "name": "agent_resource_bindings_organization_id_organizations_id_fk", + "tableFrom": "agent_resource_bindings", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "agent_resource_bindings_agent_id_agents_uuid_fk": { + "name": "agent_resource_bindings_agent_id_agents_uuid_fk", + "tableFrom": "agent_resource_bindings", + "tableTo": "agents", + "columnsFrom": [ + "agent_id" + ], + "columnsTo": [ + "uuid" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "agent_resource_bindings_resource_id_resources_id_fk": { + "name": "agent_resource_bindings_resource_id_resources_id_fk", + "tableFrom": "agent_resource_bindings", + "tableTo": "resources", + "columnsFrom": [ + "resource_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "agent_resource_bindings_replaces_resource_id_resources_id_fk": { + "name": "agent_resource_bindings_replaces_resource_id_resources_id_fk", + "tableFrom": "agent_resource_bindings", + "tableTo": "resources", + "columnsFrom": [ + "replaces_resource_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.agents": { + "name": "agents", + "schema": "", + "columns": { + "uuid": { + "name": "uuid", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "display_name": { + "name": "display_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "delegate_mention": { + "name": "delegate_mention", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "inbox_id": { + "name": "inbox_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'active'" + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "visibility": { + "name": "visibility", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'private'" + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "manager_id": { + "name": "manager_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "client_id": { + "name": "client_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "runtime_provider": { + "name": "runtime_provider", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'claude-code'" + }, + "avatar_color_token": { + "name": "avatar_color_token", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "avatar_image_data": { + "name": "avatar_image_data", + "type": "bytea", + "primaryKey": false, + "notNull": false + }, + "avatar_image_mime": { + "name": "avatar_image_mime", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "avatar_image_updated_at": { + "name": "avatar_image_updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "avatar_object_key": { + "name": "avatar_object_key", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "skills": { + "name": "skills", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'[]'::jsonb" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_agents_org": { + "name": "idx_agents_org", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_agents_manager": { + "name": "idx_agents_manager", + "columns": [ + { + "expression": "manager_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_agents_visibility_org": { + "name": "idx_agents_visibility_org", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "visibility", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_agents_client": { + "name": "idx_agents_client", + "columns": [ + { + "expression": "client_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "agents_organization_id_organizations_id_fk": { + "name": "agents_organization_id_organizations_id_fk", + "tableFrom": "agents", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "agents_client_id_clients_id_fk": { + "name": "agents_client_id_clients_id_fk", + "tableFrom": "agents", + "tableTo": "clients", + "columnsFrom": [ + "client_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "restrict", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "agents_inbox_id_unique": { + "name": "agents_inbox_id_unique", + "nullsNotDistinct": false, + "columns": [ + "inbox_id" + ] + }, + "uq_agents_org_name": { + "name": "uq_agents_org_name", + "nullsNotDistinct": false, + "columns": [ + "organization_id", + "name" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.attachment_references": { + "name": "attachment_references", + "schema": "", + "columns": { + "attachment_id": { + "name": "attachment_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "message_id": { + "name": "message_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "attachment_references_message_id_idx": { + "name": "attachment_references_message_id_idx", + "columns": [ + { + "expression": "message_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "attachment_references_attachment_id_attachments_id_fk": { + "name": "attachment_references_attachment_id_attachments_id_fk", + "tableFrom": "attachment_references", + "tableTo": "attachments", + "columnsFrom": [ + "attachment_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "attachment_references_message_id_messages_id_fk": { + "name": "attachment_references_message_id_messages_id_fk", + "tableFrom": "attachment_references", + "tableTo": "messages", + "columnsFrom": [ + "message_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "attachment_references_attachment_id_message_id_pk": { + "name": "attachment_references_attachment_id_message_id_pk", + "columns": [ + "attachment_id", + "message_id" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.attachments": { + "name": "attachments", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "mime_type": { + "name": "mime_type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "filename": { + "name": "filename", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "size_bytes": { + "name": "size_bytes", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "object_key": { + "name": "object_key", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "state": { + "name": "state", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'stored'" + }, + "data": { + "name": "data", + "type": "bytea", + "primaryKey": false, + "notNull": false + }, + "uploaded_by": { + "name": "uploaded_by", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "attachments_uploaded_by_idx": { + "name": "attachments_uploaded_by_idx", + "columns": [ + { + "expression": "uploaded_by", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "attachments_created_at_idx": { + "name": "attachments_created_at_idx", + "columns": [ + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "attachments_org_state_idx": { + "name": "attachments_org_state_idx", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "state", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "attachments_state_created_at_idx": { + "name": "attachments_state_created_at_idx", + "columns": [ + { + "expression": "state", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "attachments_organization_id_organizations_id_fk": { + "name": "attachments_organization_id_organizations_id_fk", + "tableFrom": "attachments", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.attentions": { + "name": "attentions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "origin_agent_id": { + "name": "origin_agent_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "origin_chat_id": { + "name": "origin_chat_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "target_human_id": { + "name": "target_human_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "subject": { + "name": "subject", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "body": { + "name": "body", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "''" + }, + "requires_response": { + "name": "requires_response", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "state": { + "name": "state", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'open'" + }, + "response": { + "name": "response", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "responded_by": { + "name": "responded_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "responded_at": { + "name": "responded_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "cancelled": { + "name": "cancelled", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "cancelled_reason": { + "name": "cancelled_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "closed_at": { + "name": "closed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "idx_attentions_target_open": { + "name": "idx_attentions_target_open", + "columns": [ + { + "expression": "target_human_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "state", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_attentions_chat_open": { + "name": "idx_attentions_chat_open", + "columns": [ + { + "expression": "origin_chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "state", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_attentions_origin": { + "name": "idx_attentions_origin", + "columns": [ + { + "expression": "origin_agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.auth_identities": { + "name": "auth_identities", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "identifier": { + "name": "identifier", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "email": { + "name": "email", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "verified_at": { + "name": "verified_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "credential_type": { + "name": "credential_type", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "credential_payload": { + "name": "credential_payload", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_auth_identities_user": { + "name": "idx_auth_identities_user", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_auth_identities_email": { + "name": "idx_auth_identities_email", + "columns": [ + { + "expression": "email", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "auth_identities_user_id_users_id_fk": { + "name": "auth_identities_user_id_users_id_fk", + "tableFrom": "auth_identities", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "uq_auth_identities_provider_identifier": { + "name": "uq_auth_identities_provider_identifier", + "nullsNotDistinct": false, + "columns": [ + "provider", + "identifier" + ] + }, + "uq_auth_identities_user_provider": { + "name": "uq_auth_identities_user_provider", + "nullsNotDistinct": false, + "columns": [ + "user_id", + "provider" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.chat_membership": { + "name": "chat_membership", + "schema": "", + "columns": { + "chat_id": { + "name": "chat_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'member'" + }, + "access_mode": { + "name": "access_mode", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "mode": { + "name": "mode", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'full'" + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'manual'" + }, + "joined_at": { + "name": "joined_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_membership_agent": { + "name": "idx_membership_agent", + "columns": [ + { + "expression": "agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_membership_chat_role": { + "name": "idx_membership_chat_role", + "columns": [ + { + "expression": "chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "access_mode", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": { + "chat_membership_chat_id_agent_id_pk": { + "name": "chat_membership_chat_id_agent_id_pk", + "columns": [ + "chat_id", + "agent_id" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.chat_user_state": { + "name": "chat_user_state", + "schema": "", + "columns": { + "chat_id": { + "name": "chat_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "last_read_at": { + "name": "last_read_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "unread_mention_count": { + "name": "unread_mention_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "open_request_count": { + "name": "open_request_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "engagement_status": { + "name": "engagement_status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'active'" + }, + "pinned_at": { + "name": "pinned_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "idx_user_state_agent": { + "name": "idx_user_state_agent", + "columns": [ + { + "expression": "agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_user_state_unread": { + "name": "idx_user_state_unread", + "columns": [ + { + "expression": "agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "unread_mention_count > 0", + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_user_state_open_req": { + "name": "idx_user_state_open_req", + "columns": [ + { + "expression": "agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "open_request_count > 0", + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_user_state_pinned": { + "name": "idx_user_state_pinned", + "columns": [ + { + "expression": "agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "pinned_at IS NOT NULL", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": { + "chat_user_state_chat_id_agent_id_pk": { + "name": "chat_user_state_chat_id_agent_id_pk", + "columns": [ + "chat_id", + "agent_id" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.chats": { + "name": "chats", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'direct'" + }, + "topic": { + "name": "topic", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "description_updated_at": { + "name": "description_updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "lifecycle_policy": { + "name": "lifecycle_policy", + "type": "text", + "primaryKey": false, + "notNull": false, + "default": "'persistent'" + }, + "parent_chat_id": { + "name": "parent_chat_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "onboarding_kickoff_key": { + "name": "onboarding_kickoff_key", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "last_message_at": { + "name": "last_message_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_message_preview": { + "name": "last_message_preview", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "activity_at": { + "name": "activity_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_chats_org_last_message": { + "name": "idx_chats_org_last_message", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "\"last_message_at\" desc", + "asc": true, + "isExpression": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_chats_org_activity": { + "name": "idx_chats_org_activity", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "\"activity_at\" desc", + "asc": true, + "isExpression": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "uq_chats_onboarding_kickoff_key": { + "name": "uq_chats_onboarding_kickoff_key", + "columns": [ + { + "expression": "onboarding_kickoff_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "chats_organization_id_organizations_id_fk": { + "name": "chats_organization_id_organizations_id_fk", + "tableFrom": "chats", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.clients": { + "name": "clients", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'disconnected'" + }, + "sdk_version": { + "name": "sdk_version", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "hostname": { + "name": "hostname", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "os": { + "name": "os", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "instance_id": { + "name": "instance_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "connected_at": { + "name": "connected_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "retired_at": { + "name": "retired_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "paused_reason": { + "name": "paused_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "idx_clients_user": { + "name": "idx_clients_user", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_clients_org": { + "name": "idx_clients_org", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "clients_user_id_users_id_fk": { + "name": "clients_user_id_users_id_fk", + "tableFrom": "clients", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "clients_organization_id_organizations_id_fk": { + "name": "clients_organization_id_organizations_id_fk", + "tableFrom": "clients", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "ck_clients_paused_reason": { + "name": "ck_clients_paused_reason", + "value": "\"clients\".\"paused_reason\" IS NULL OR \"clients\".\"paused_reason\" IN ('auth_rejected', 'auth_refresh_failed')" + } + }, + "isRLSEnabled": false + }, + "public.connect_codes": { + "name": "connect_codes", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "code_hash": { + "name": "code_hash", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "issuer": { + "name": "issuer", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "consumed_at": { + "name": "consumed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_connect_codes_user": { + "name": "idx_connect_codes_user", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_connect_codes_expires_at": { + "name": "idx_connect_codes_expires_at", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "connect_codes_user_id_users_id_fk": { + "name": "connect_codes_user_id_users_id_fk", + "tableFrom": "connect_codes", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "connect_codes_code_hash_unique": { + "name": "connect_codes_code_hash_unique", + "nullsNotDistinct": false, + "columns": [ + "code_hash" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.context_tree_io_events": { + "name": "context_tree_io_events", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "chat_id": { + "name": "chat_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "source_session_event_id": { + "name": "source_session_event_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "source_index": { + "name": "source_index", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "runtime_provider": { + "name": "runtime_provider", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "action": { + "name": "action", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "tree_repo_url": { + "name": "tree_repo_url", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "tree_branch": { + "name": "tree_branch", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "target_kind": { + "name": "target_kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "target_path": { + "name": "target_path", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "uq_context_tree_io_source": { + "name": "uq_context_tree_io_source", + "columns": [ + { + "expression": "source_session_event_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "source_index", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_context_tree_io_org_recent": { + "name": "idx_context_tree_io_org_recent", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": false, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_context_tree_io_org_action_recent": { + "name": "idx_context_tree_io_org_action_recent", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "action", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": false, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_context_tree_io_org_agent_recent": { + "name": "idx_context_tree_io_org_agent_recent", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": false, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_context_tree_io_org_target_recent": { + "name": "idx_context_tree_io_org_target_recent", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "target_path", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": false, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "context_tree_io_events_organization_id_organizations_id_fk": { + "name": "context_tree_io_events_organization_id_organizations_id_fk", + "tableFrom": "context_tree_io_events", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "context_tree_io_events_agent_id_agents_uuid_fk": { + "name": "context_tree_io_events_agent_id_agents_uuid_fk", + "tableFrom": "context_tree_io_events", + "tableTo": "agents", + "columnsFrom": [ + "agent_id" + ], + "columnsTo": [ + "uuid" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "context_tree_io_events_chat_id_chats_id_fk": { + "name": "context_tree_io_events_chat_id_chats_id_fk", + "tableFrom": "context_tree_io_events", + "tableTo": "chats", + "columnsFrom": [ + "chat_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "ck_context_tree_io_action": { + "name": "ck_context_tree_io_action", + "value": "\"context_tree_io_events\".\"action\" IN ('read', 'write')" + }, + "ck_context_tree_io_target_kind": { + "name": "ck_context_tree_io_target_kind", + "value": "\"context_tree_io_events\".\"target_kind\" IN ('file', 'directory', 'repo')" + }, + "ck_context_tree_io_target_path_nonempty": { + "name": "ck_context_tree_io_target_path_nonempty", + "value": "\"context_tree_io_events\".\"target_path\" <> ''" + } + }, + "isRLSEnabled": false + }, + "public.cron_jobs": { + "name": "cron_jobs", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "owner_member_id": { + "name": "owner_member_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "control_chat_id": { + "name": "control_chat_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "chat_mode": { + "name": "chat_mode", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'reuse_control_chat'" + }, + "cron_expression": { + "name": "cron_expression", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "timezone": { + "name": "timezone", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "prompt": { + "name": "prompt", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "state": { + "name": "state", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "state_reason": { + "name": "state_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "revision": { + "name": "revision", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 1 + }, + "next_run_at": { + "name": "next_run_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_trigger_message_id": { + "name": "last_trigger_message_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_cron_jobs_due": { + "name": "idx_cron_jobs_due", + "columns": [ + { + "expression": "next_run_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"cron_jobs\".\"state\" = 'active'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_cron_jobs_control_created": { + "name": "idx_cron_jobs_control_created", + "columns": [ + { + "expression": "control_chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_cron_jobs_owner_created": { + "name": "idx_cron_jobs_owner_created", + "columns": [ + { + "expression": "owner_member_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "cron_jobs_owner_member_id_members_id_fk": { + "name": "cron_jobs_owner_member_id_members_id_fk", + "tableFrom": "cron_jobs", + "tableTo": "members", + "columnsFrom": [ + "owner_member_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "cron_jobs_control_chat_id_chats_id_fk": { + "name": "cron_jobs_control_chat_id_chats_id_fk", + "tableFrom": "cron_jobs", + "tableTo": "chats", + "columnsFrom": [ + "control_chat_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "cron_jobs_agent_id_agents_uuid_fk": { + "name": "cron_jobs_agent_id_agents_uuid_fk", + "tableFrom": "cron_jobs", + "tableTo": "agents", + "columnsFrom": [ + "agent_id" + ], + "columnsTo": [ + "uuid" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "cron_jobs_last_trigger_message_id_messages_id_fk": { + "name": "cron_jobs_last_trigger_message_id_messages_id_fk", + "tableFrom": "cron_jobs", + "tableTo": "messages", + "columnsFrom": [ + "last_trigger_message_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "uq_cron_jobs_control_agent_name": { + "name": "uq_cron_jobs_control_agent_name", + "nullsNotDistinct": false, + "columns": [ + "control_chat_id", + "agent_id", + "name" + ] + } + }, + "policies": {}, + "checkConstraints": { + "ck_cron_jobs_state": { + "name": "ck_cron_jobs_state", + "value": "\"cron_jobs\".\"state\" IN ('active', 'paused')" + }, + "ck_cron_jobs_chat_mode": { + "name": "ck_cron_jobs_chat_mode", + "value": "\"cron_jobs\".\"chat_mode\" = 'reuse_control_chat'" + }, + "ck_cron_jobs_revision_positive": { + "name": "ck_cron_jobs_revision_positive", + "value": "\"cron_jobs\".\"revision\" > 0" + }, + "ck_cron_jobs_active_shape": { + "name": "ck_cron_jobs_active_shape", + "value": "(\"cron_jobs\".\"state\" = 'active' AND \"cron_jobs\".\"next_run_at\" IS NOT NULL AND \"cron_jobs\".\"state_reason\" IS NULL) OR (\"cron_jobs\".\"state\" = 'paused' AND \"cron_jobs\".\"next_run_at\" IS NULL AND \"cron_jobs\".\"state_reason\" IS NOT NULL)" + } + }, + "isRLSEnabled": false + }, + "public.doc_comments": { + "name": "doc_comments", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "document_id": { + "name": "document_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "version_number": { + "name": "version_number", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "parent_id": { + "name": "parent_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "author_kind": { + "name": "author_kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "author_id": { + "name": "author_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "author_name": { + "name": "author_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "body": { + "name": "body", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "anchor": { + "name": "anchor", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'open'" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "doc_comments_document_status_idx": { + "name": "doc_comments_document_status_idx", + "columns": [ + { + "expression": "document_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "doc_comments_parent_idx": { + "name": "doc_comments_parent_idx", + "columns": [ + { + "expression": "parent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "doc_comments_document_id_doc_documents_id_fk": { + "name": "doc_comments_document_id_doc_documents_id_fk", + "tableFrom": "doc_comments", + "tableTo": "doc_documents", + "columnsFrom": [ + "document_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "doc_comments_parent_id_doc_comments_id_fk": { + "name": "doc_comments_parent_id_doc_comments_id_fk", + "tableFrom": "doc_comments", + "tableTo": "doc_comments", + "columnsFrom": [ + "parent_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.doc_documents": { + "name": "doc_documents", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "slug": { + "name": "slug", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "project": { + "name": "project", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'draft'" + }, + "latest_version": { + "name": "latest_version", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "created_by_kind": { + "name": "created_by_kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_by_id": { + "name": "created_by_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_by_name": { + "name": "created_by_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "doc_documents_org_slug_unique": { + "name": "doc_documents_org_slug_unique", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "slug", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "doc_documents_org_updated_idx": { + "name": "doc_documents_org_updated_idx", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "doc_documents_organization_id_organizations_id_fk": { + "name": "doc_documents_organization_id_organizations_id_fk", + "tableFrom": "doc_documents", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.doc_versions": { + "name": "doc_versions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "document_id": { + "name": "document_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "number": { + "name": "number", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "content": { + "name": "content", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "note": { + "name": "note", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "author_kind": { + "name": "author_kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "author_id": { + "name": "author_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "author_name": { + "name": "author_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "doc_versions_document_number_unique": { + "name": "doc_versions_document_number_unique", + "columns": [ + { + "expression": "document_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "number", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "doc_versions_document_id_doc_documents_id_fk": { + "name": "doc_versions_document_id_doc_documents_id_fk", + "tableFrom": "doc_versions", + "tableTo": "doc_documents", + "columnsFrom": [ + "document_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.github_app_installations": { + "name": "github_app_installations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "installation_id": { + "name": "installation_id", + "type": "bigint", + "primaryKey": false, + "notNull": true + }, + "account_type": { + "name": "account_type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "account_login": { + "name": "account_login", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "account_github_id": { + "name": "account_github_id", + "type": "bigint", + "primaryKey": false, + "notNull": true + }, + "installer_github_id": { + "name": "installer_github_id", + "type": "bigint", + "primaryKey": false, + "notNull": false + }, + "requester_github_id": { + "name": "requester_github_id", + "type": "bigint", + "primaryKey": false, + "notNull": false + }, + "hub_organization_id": { + "name": "hub_organization_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "permissions": { + "name": "permissions", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "events": { + "name": "events", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "suspended_at": { + "name": "suspended_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "uq_github_app_installations_installation_id": { + "name": "uq_github_app_installations_installation_id", + "columns": [ + { + "expression": "installation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "uq_github_app_installations_hub_org": { + "name": "uq_github_app_installations_hub_org", + "columns": [ + { + "expression": "hub_organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_github_app_installations_account": { + "name": "idx_github_app_installations_account", + "columns": [ + { + "expression": "account_github_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_github_app_installations_installer": { + "name": "idx_github_app_installations_installer", + "columns": [ + { + "expression": "installer_github_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_github_app_installations_requester": { + "name": "idx_github_app_installations_requester", + "columns": [ + { + "expression": "requester_github_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "github_app_installations_hub_organization_id_organizations_id_fk": { + "name": "github_app_installations_hub_organization_id_organizations_id_fk", + "tableFrom": "github_app_installations", + "tableTo": "organizations", + "columnsFrom": [ + "hub_organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "ck_github_app_installations_account_type": { + "name": "ck_github_app_installations_account_type", + "value": "\"github_app_installations\".\"account_type\" IN ('User', 'Organization')" + } + }, + "isRLSEnabled": false + }, + "public.github_entity_chat_mappings": { + "name": "github_entity_chat_mappings", + "schema": "", + "columns": { + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "human_agent_id": { + "name": "human_agent_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "delegate_agent_id": { + "name": "delegate_agent_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "entity_type": { + "name": "entity_type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "entity_key": { + "name": "entity_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "chat_id": { + "name": "chat_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "bound_at": { + "name": "bound_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "bound_via": { + "name": "bound_via", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "entity_state": { + "name": "entity_state", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'open'" + }, + "entity_state_updated_at": { + "name": "entity_state_updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "idx_github_entity_chat_mappings_chat": { + "name": "idx_github_entity_chat_mappings_chat", + "columns": [ + { + "expression": "chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_github_entity_chat_mappings_chat_state": { + "name": "idx_github_entity_chat_mappings_chat_state", + "columns": [ + { + "expression": "chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_state", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "github_entity_chat_mappings_organization_id_organizations_id_fk": { + "name": "github_entity_chat_mappings_organization_id_organizations_id_fk", + "tableFrom": "github_entity_chat_mappings", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "github_entity_chat_mappings_human_agent_id_agents_uuid_fk": { + "name": "github_entity_chat_mappings_human_agent_id_agents_uuid_fk", + "tableFrom": "github_entity_chat_mappings", + "tableTo": "agents", + "columnsFrom": [ + "human_agent_id" + ], + "columnsTo": [ + "uuid" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "github_entity_chat_mappings_delegate_agent_id_agents_uuid_fk": { + "name": "github_entity_chat_mappings_delegate_agent_id_agents_uuid_fk", + "tableFrom": "github_entity_chat_mappings", + "tableTo": "agents", + "columnsFrom": [ + "delegate_agent_id" + ], + "columnsTo": [ + "uuid" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "github_entity_chat_mappings_chat_id_chats_id_fk": { + "name": "github_entity_chat_mappings_chat_id_chats_id_fk", + "tableFrom": "github_entity_chat_mappings", + "tableTo": "chats", + "columnsFrom": [ + "chat_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "github_entity_chat_mappings_organization_id_human_agent_id_delegate_agent_id_entity_type_entity_key_pk": { + "name": "github_entity_chat_mappings_organization_id_human_agent_id_delegate_agent_id_entity_type_entity_key_pk", + "columns": [ + "organization_id", + "human_agent_id", + "delegate_agent_id", + "entity_type", + "entity_key" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.gitlab_connections": { + "name": "gitlab_connections", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "display_name": { + "name": "display_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "instance_origin": { + "name": "instance_origin", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "token_hash": { + "name": "token_hash", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "endpoint_first_seen_at": { + "name": "endpoint_first_seen_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_valid_inbound_at": { + "name": "last_valid_inbound_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_processing_failure_at": { + "name": "last_processing_failure_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_processing_failure_code": { + "name": "last_processing_failure_code", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "stable_delivery_observed_at": { + "name": "stable_delivery_observed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_observed_version": { + "name": "last_observed_version", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "reviewer_mode": { + "name": "reviewer_mode", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'unknown'" + }, + "last_reviewer_schema_anomaly_at": { + "name": "last_reviewer_schema_anomaly_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_reviewer_schema_anomaly_code": { + "name": "last_reviewer_schema_anomaly_code", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_by_member_id": { + "name": "created_by_member_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "updated_by_member_id": { + "name": "updated_by_member_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "uq_gitlab_connections_org": { + "name": "uq_gitlab_connections_org", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "uq_gitlab_connections_token_hash": { + "name": "uq_gitlab_connections_token_hash", + "columns": [ + { + "expression": "token_hash", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "gitlab_connections_organization_id_organizations_id_fk": { + "name": "gitlab_connections_organization_id_organizations_id_fk", + "tableFrom": "gitlab_connections", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "gitlab_connections_created_by_member_id_members_id_fk": { + "name": "gitlab_connections_created_by_member_id_members_id_fk", + "tableFrom": "gitlab_connections", + "tableTo": "members", + "columnsFrom": [ + "created_by_member_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "gitlab_connections_updated_by_member_id_members_id_fk": { + "name": "gitlab_connections_updated_by_member_id_members_id_fk", + "tableFrom": "gitlab_connections", + "tableTo": "members", + "columnsFrom": [ + "updated_by_member_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "ck_gitlab_connections_reviewer_mode": { + "name": "ck_gitlab_connections_reviewer_mode", + "value": "\"gitlab_connections\".\"reviewer_mode\" IN ('unknown', 'assignee', 'reviewers')" + } + }, + "isRLSEnabled": false + }, + "public.gitlab_entity_chat_mappings": { + "name": "gitlab_entity_chat_mappings", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "chat_id": { + "name": "chat_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "declared_by_agent_id": { + "name": "declared_by_agent_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "bound_via": { + "name": "bound_via", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'agent_declared'" + }, + "identity_link_id": { + "name": "identity_link_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "human_agent_id": { + "name": "human_agent_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "delegate_agent_id": { + "name": "delegate_agent_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "attention_mode": { + "name": "attention_mode", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'legacy_route_only'" + }, + "attention_backfill_version": { + "name": "attention_backfill_version", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "active": { + "name": "active", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": true + }, + "entity_type": { + "name": "entity_type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "entity_iid": { + "name": "entity_iid", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "bigint", + "primaryKey": false, + "notNull": false + }, + "project_path": { + "name": "project_path", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "project_path_normalized": { + "name": "project_path_normalized", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "entity_url": { + "name": "entity_url", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "entity_state": { + "name": "entity_state", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'open'" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "uq_gitlab_entity_pending_pair": { + "name": "uq_gitlab_entity_pending_pair", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "human_agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "delegate_agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "project_path_normalized", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_type", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_iid", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"gitlab_entity_chat_mappings\".\"project_id\" IS NULL AND \"gitlab_entity_chat_mappings\".\"active\" AND \"gitlab_entity_chat_mappings\".\"bound_via\" <> 'identity_target' AND \"gitlab_entity_chat_mappings\".\"human_agent_id\" IS NOT NULL AND \"gitlab_entity_chat_mappings\".\"delegate_agent_id\" IS NOT NULL", + "concurrently": false, + "method": "btree", + "with": {} + }, + "uq_gitlab_entity_observed_pair": { + "name": "uq_gitlab_entity_observed_pair", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "human_agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "delegate_agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_type", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_iid", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"gitlab_entity_chat_mappings\".\"project_id\" IS NOT NULL AND \"gitlab_entity_chat_mappings\".\"active\" AND \"gitlab_entity_chat_mappings\".\"bound_via\" <> 'identity_target' AND \"gitlab_entity_chat_mappings\".\"human_agent_id\" IS NOT NULL AND \"gitlab_entity_chat_mappings\".\"delegate_agent_id\" IS NOT NULL", + "concurrently": false, + "method": "btree", + "with": {} + }, + "uq_gitlab_entity_pending_legacy_chat": { + "name": "uq_gitlab_entity_pending_legacy_chat", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "project_path_normalized", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_type", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_iid", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"gitlab_entity_chat_mappings\".\"project_id\" IS NULL AND \"gitlab_entity_chat_mappings\".\"active\" AND \"gitlab_entity_chat_mappings\".\"bound_via\" <> 'identity_target' AND \"gitlab_entity_chat_mappings\".\"human_agent_id\" IS NULL AND \"gitlab_entity_chat_mappings\".\"delegate_agent_id\" IS NULL", + "concurrently": false, + "method": "btree", + "with": {} + }, + "uq_gitlab_entity_observed_legacy_chat": { + "name": "uq_gitlab_entity_observed_legacy_chat", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_type", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_iid", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"gitlab_entity_chat_mappings\".\"project_id\" IS NOT NULL AND \"gitlab_entity_chat_mappings\".\"active\" AND \"gitlab_entity_chat_mappings\".\"bound_via\" <> 'identity_target' AND \"gitlab_entity_chat_mappings\".\"human_agent_id\" IS NULL AND \"gitlab_entity_chat_mappings\".\"delegate_agent_id\" IS NULL", + "concurrently": false, + "method": "btree", + "with": {} + }, + "uq_gitlab_entity_identity_target": { + "name": "uq_gitlab_entity_identity_target", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "identity_link_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_type", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_iid", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"gitlab_entity_chat_mappings\".\"project_id\" IS NOT NULL AND \"gitlab_entity_chat_mappings\".\"active\" AND \"gitlab_entity_chat_mappings\".\"bound_via\" = 'identity_target'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_gitlab_entity_observed_lookup": { + "name": "idx_gitlab_entity_observed_lookup", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_type", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_iid", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_gitlab_entity_pending_lookup": { + "name": "idx_gitlab_entity_pending_lookup", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "project_path_normalized", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_type", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_iid", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"gitlab_entity_chat_mappings\".\"project_id\" IS NULL", + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_gitlab_entity_chat": { + "name": "idx_gitlab_entity_chat", + "columns": [ + { + "expression": "chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "gitlab_entity_chat_mappings_organization_id_organizations_id_fk": { + "name": "gitlab_entity_chat_mappings_organization_id_organizations_id_fk", + "tableFrom": "gitlab_entity_chat_mappings", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "gitlab_entity_chat_mappings_connection_id_gitlab_connections_id_fk": { + "name": "gitlab_entity_chat_mappings_connection_id_gitlab_connections_id_fk", + "tableFrom": "gitlab_entity_chat_mappings", + "tableTo": "gitlab_connections", + "columnsFrom": [ + "connection_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "gitlab_entity_chat_mappings_chat_id_chats_id_fk": { + "name": "gitlab_entity_chat_mappings_chat_id_chats_id_fk", + "tableFrom": "gitlab_entity_chat_mappings", + "tableTo": "chats", + "columnsFrom": [ + "chat_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "gitlab_entity_chat_mappings_declared_by_agent_id_agents_uuid_fk": { + "name": "gitlab_entity_chat_mappings_declared_by_agent_id_agents_uuid_fk", + "tableFrom": "gitlab_entity_chat_mappings", + "tableTo": "agents", + "columnsFrom": [ + "declared_by_agent_id" + ], + "columnsTo": [ + "uuid" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "gitlab_entity_chat_mappings_identity_link_id_gitlab_identity_links_id_fk": { + "name": "gitlab_entity_chat_mappings_identity_link_id_gitlab_identity_links_id_fk", + "tableFrom": "gitlab_entity_chat_mappings", + "tableTo": "gitlab_identity_links", + "columnsFrom": [ + "identity_link_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "gitlab_entity_chat_mappings_human_agent_id_agents_uuid_fk": { + "name": "gitlab_entity_chat_mappings_human_agent_id_agents_uuid_fk", + "tableFrom": "gitlab_entity_chat_mappings", + "tableTo": "agents", + "columnsFrom": [ + "human_agent_id" + ], + "columnsTo": [ + "uuid" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "gitlab_entity_chat_mappings_delegate_agent_id_agents_uuid_fk": { + "name": "gitlab_entity_chat_mappings_delegate_agent_id_agents_uuid_fk", + "tableFrom": "gitlab_entity_chat_mappings", + "tableTo": "agents", + "columnsFrom": [ + "delegate_agent_id" + ], + "columnsTo": [ + "uuid" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "ck_gitlab_entity_type": { + "name": "ck_gitlab_entity_type", + "value": "\"gitlab_entity_chat_mappings\".\"entity_type\" IN ('issue', 'pull_request')" + }, + "ck_gitlab_entity_bound_via": { + "name": "ck_gitlab_entity_bound_via", + "value": "\"gitlab_entity_chat_mappings\".\"bound_via\" IN ('agent_declared', 'human_declared', 'identity_target')" + }, + "ck_gitlab_entity_identity_owner": { + "name": "ck_gitlab_entity_identity_owner", + "value": "\"gitlab_entity_chat_mappings\".\"bound_via\" <> 'identity_target' OR (\"gitlab_entity_chat_mappings\".\"identity_link_id\" IS NOT NULL AND \"gitlab_entity_chat_mappings\".\"human_agent_id\" IS NOT NULL AND \"gitlab_entity_chat_mappings\".\"delegate_agent_id\" IS NOT NULL AND \"gitlab_entity_chat_mappings\".\"project_id\" IS NOT NULL)" + }, + "ck_gitlab_entity_attention_pair": { + "name": "ck_gitlab_entity_attention_pair", + "value": "(\"gitlab_entity_chat_mappings\".\"human_agent_id\" IS NULL AND \"gitlab_entity_chat_mappings\".\"delegate_agent_id\" IS NULL) OR (\"gitlab_entity_chat_mappings\".\"human_agent_id\" IS NOT NULL AND \"gitlab_entity_chat_mappings\".\"delegate_agent_id\" IS NOT NULL)" + }, + "ck_gitlab_entity_attention_mode": { + "name": "ck_gitlab_entity_attention_mode", + "value": "\"gitlab_entity_chat_mappings\".\"attention_mode\" IN ('paired', 'legacy_route_only')" + } + }, + "isRLSEnabled": false + }, + "public.gitlab_identity_links": { + "name": "gitlab_identity_links", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "membership_id": { + "name": "membership_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "display_username": { + "name": "display_username", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "normalized_username": { + "name": "normalized_username", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "state": { + "name": "state", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "uq_gitlab_identity_connection_membership": { + "name": "uq_gitlab_identity_connection_membership", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "membership_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "uq_gitlab_identity_connection_username": { + "name": "uq_gitlab_identity_connection_username", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "normalized_username", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_gitlab_identity_org_state": { + "name": "idx_gitlab_identity_org_state", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "state", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_gitlab_identity_membership_state": { + "name": "idx_gitlab_identity_membership_state", + "columns": [ + { + "expression": "membership_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "state", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "gitlab_identity_links_organization_id_organizations_id_fk": { + "name": "gitlab_identity_links_organization_id_organizations_id_fk", + "tableFrom": "gitlab_identity_links", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "gitlab_identity_links_membership_id_members_id_fk": { + "name": "gitlab_identity_links_membership_id_members_id_fk", + "tableFrom": "gitlab_identity_links", + "tableTo": "members", + "columnsFrom": [ + "membership_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "restrict", + "onUpdate": "no action" + }, + "gitlab_identity_links_connection_id_gitlab_connections_id_fk": { + "name": "gitlab_identity_links_connection_id_gitlab_connections_id_fk", + "tableFrom": "gitlab_identity_links", + "tableTo": "gitlab_connections", + "columnsFrom": [ + "connection_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "ck_gitlab_identity_state": { + "name": "ck_gitlab_identity_state", + "value": "\"gitlab_identity_links\".\"state\" IN ('active', 'suspended')" + } + }, + "isRLSEnabled": false + }, + "public.inbox_entries": { + "name": "inbox_entries", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "bigserial", + "primaryKey": true, + "notNull": true + }, + "inbox_id": { + "name": "inbox_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "message_id": { + "name": "message_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "chat_id": { + "name": "chat_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "notify": { + "name": "notify", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": true + }, + "retry_count": { + "name": "retry_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "delivered_at": { + "name": "delivered_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "acked_at": { + "name": "acked_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "idx_inbox_pending": { + "name": "idx_inbox_pending", + "columns": [ + { + "expression": "inbox_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_inbox_pending_notify": { + "name": "idx_inbox_pending_notify", + "columns": [ + { + "expression": "inbox_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "status = 'pending' AND notify = true", + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_inbox_chat_silent": { + "name": "idx_inbox_chat_silent", + "columns": [ + { + "expression": "inbox_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "notify", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_inbox_entries_message_status": { + "name": "idx_inbox_entries_message_status", + "columns": [ + { + "expression": "message_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "inbox_entries_message_id_messages_id_fk": { + "name": "inbox_entries_message_id_messages_id_fk", + "tableFrom": "inbox_entries", + "tableTo": "messages", + "columnsFrom": [ + "message_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "uq_inbox_delivery": { + "name": "uq_inbox_delivery", + "nullsNotDistinct": false, + "columns": [ + "inbox_id", + "message_id", + "chat_id" + ] + } + }, + "policies": {}, + "checkConstraints": { + "ck_inbox_entries_status": { + "name": "ck_inbox_entries_status", + "value": "\"inbox_entries\".\"status\" IN ('pending', 'delivered', 'acked')" + } + }, + "isRLSEnabled": false + }, + "public.invitation_redemptions": { + "name": "invitation_redemptions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "invitation_id": { + "name": "invitation_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "redeemed_at": { + "name": "redeemed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "ip": { + "name": "ip", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "user_agent": { + "name": "user_agent", + "type": "text", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "idx_invitation_redemptions_invitation": { + "name": "idx_invitation_redemptions_invitation", + "columns": [ + { + "expression": "invitation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_invitation_redemptions_user": { + "name": "idx_invitation_redemptions_user", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "invitation_redemptions_invitation_id_invitations_id_fk": { + "name": "invitation_redemptions_invitation_id_invitations_id_fk", + "tableFrom": "invitation_redemptions", + "tableTo": "invitations", + "columnsFrom": [ + "invitation_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "invitation_redemptions_user_id_users_id_fk": { + "name": "invitation_redemptions_user_id_users_id_fk", + "tableFrom": "invitation_redemptions", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.invitations": { + "name": "invitations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'member'" + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "revoked_at": { + "name": "revoked_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_invitations_token": { + "name": "idx_invitations_token", + "columns": [ + { + "expression": "token", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_invitations_org": { + "name": "idx_invitations_org", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "invitations_organization_id_organizations_id_fk": { + "name": "invitations_organization_id_organizations_id_fk", + "tableFrom": "invitations", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "invitations_created_by_users_id_fk": { + "name": "invitations_created_by_users_id_fk", + "tableFrom": "invitations", + "tableTo": "users", + "columnsFrom": [ + "created_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "invitations_token_unique": { + "name": "invitations_token_unique", + "nullsNotDistinct": false, + "columns": [ + "token" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.members": { + "name": "members", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "onboarding_suppressed_at": { + "name": "onboarding_suppressed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "onboarding_suppressed_reason": { + "name": "onboarding_suppressed_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "onboarding_completed_at": { + "name": "onboarding_completed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'active'" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_members_user": { + "name": "idx_members_user", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_members_org": { + "name": "idx_members_org", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "members_user_id_users_id_fk": { + "name": "members_user_id_users_id_fk", + "tableFrom": "members", + "tableTo": "users", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "members_organization_id_organizations_id_fk": { + "name": "members_organization_id_organizations_id_fk", + "tableFrom": "members", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "members_agent_id_agents_uuid_fk": { + "name": "members_agent_id_agents_uuid_fk", + "tableFrom": "members", + "tableTo": "agents", + "columnsFrom": [ + "agent_id" + ], + "columnsTo": [ + "uuid" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "members_agent_id_unique": { + "name": "members_agent_id_unique", + "nullsNotDistinct": false, + "columns": [ + "agent_id" + ] + }, + "uq_members_user_org": { + "name": "uq_members_user_org", + "nullsNotDistinct": false, + "columns": [ + "user_id", + "organization_id" + ] + } + }, + "policies": {}, + "checkConstraints": { + "ck_members_completed_implies_suppressed": { + "name": "ck_members_completed_implies_suppressed", + "value": "\"members\".\"onboarding_completed_at\" IS NULL OR \"members\".\"onboarding_suppressed_at\" IS NOT NULL" + } + }, + "isRLSEnabled": false + }, + "public.messages": { + "name": "messages", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "chat_id": { + "name": "chat_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "sender_id": { + "name": "sender_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "format": { + "name": "format", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "content": { + "name": "content", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "reply_to_inbox": { + "name": "reply_to_inbox", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "reply_to_chat": { + "name": "reply_to_chat", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "in_reply_to": { + "name": "in_reply_to", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_messages_chat_time": { + "name": "idx_messages_chat_time", + "columns": [ + { + "expression": "chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_messages_in_reply_to": { + "name": "idx_messages_in_reply_to", + "columns": [ + { + "expression": "in_reply_to", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_messages_chat_source_time": { + "name": "idx_messages_chat_source_time", + "columns": [ + { + "expression": "chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "source", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": false, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_messages_mentions": { + "name": "idx_messages_mentions", + "columns": [ + { + "expression": "((\"metadata\" -> 'mentions')) jsonb_path_ops", + "asc": true, + "isExpression": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "gin", + "with": {} + } + }, + "foreignKeys": { + "messages_chat_id_chats_id_fk": { + "name": "messages_chat_id_chats_id_fk", + "tableFrom": "messages", + "tableTo": "chats", + "columnsFrom": [ + "chat_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.notifications": { + "name": "notifications", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "severity": { + "name": "severity", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "chat_id": { + "name": "chat_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "message": { + "name": "message", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "read": { + "name": "read", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "dedup_key": { + "name": "dedup_key", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_notifications_org_created": { + "name": "idx_notifications_org_created", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_notifications_agent": { + "name": "idx_notifications_agent", + "columns": [ + { + "expression": "agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_notifications_org_read": { + "name": "idx_notifications_org_read", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "read", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "uq_notifications_org_dedup_unread": { + "name": "uq_notifications_org_dedup_unread", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "dedup_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "read = false AND dedup_key IS NOT NULL", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.organization_settings": { + "name": "organization_settings", + "schema": "", + "columns": { + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "namespace": { + "name": "namespace", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "value": { + "name": "value", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "version": { + "name": "version", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_org_settings_namespace": { + "name": "idx_org_settings_namespace", + "columns": [ + { + "expression": "namespace", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "organization_settings_organization_id_organizations_id_fk": { + "name": "organization_settings_organization_id_organizations_id_fk", + "tableFrom": "organization_settings", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "organization_settings_updated_by_users_id_fk": { + "name": "organization_settings_updated_by_users_id_fk", + "tableFrom": "organization_settings", + "tableTo": "users", + "columnsFrom": [ + "updated_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "organization_settings_organization_id_namespace_pk": { + "name": "organization_settings_organization_id_namespace_pk", + "columns": [ + "organization_id", + "namespace" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.organizations": { + "name": "organizations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "display_name": { + "name": "display_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "max_agents": { + "name": "max_agents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "max_messages_per_minute": { + "name": "max_messages_per_minute", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "features": { + "name": "features", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "organizations_name_unique": { + "name": "organizations_name_unique", + "nullsNotDistinct": false, + "columns": [ + "name" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.pending_questions": { + "name": "pending_questions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "chat_id": { + "name": "chat_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "message_id": { + "name": "message_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "answered_at": { + "name": "answered_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "superseded_at": { + "name": "superseded_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "superseded_reason": { + "name": "superseded_reason", + "type": "text", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "idx_pending_questions_agent_status": { + "name": "idx_pending_questions_agent_status", + "columns": [ + { + "expression": "agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_pending_questions_chat_status": { + "name": "idx_pending_questions_chat_status", + "columns": [ + { + "expression": "chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.resources": { + "name": "resources", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "scope": { + "name": "scope", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "owner_agent_id": { + "name": "owner_agent_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "repo_canonical_key": { + "name": "repo_canonical_key", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "default_enabled": { + "name": "default_enabled", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'active'" + }, + "payload": { + "name": "payload", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "idx_resources_org_type_scope": { + "name": "idx_resources_org_type_scope", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "type", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "scope", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_resources_owner_agent": { + "name": "idx_resources_owner_agent", + "columns": [ + { + "expression": "owner_agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_resources_repo_key": { + "name": "idx_resources_repo_key", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "repo_canonical_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "uq_resources_team_repo_canonical_active": { + "name": "uq_resources_team_repo_canonical_active", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "repo_canonical_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"resources\".\"type\" = 'repo' AND \"resources\".\"scope\" = 'team' AND \"resources\".\"status\" IN ('active', 'stale') AND \"resources\".\"repo_canonical_key\" IS NOT NULL", + "concurrently": false, + "method": "btree", + "with": {} + }, + "uq_resources_agent_repo_canonical_active": { + "name": "uq_resources_agent_repo_canonical_active", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "owner_agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "repo_canonical_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"resources\".\"type\" = 'repo' AND \"resources\".\"scope\" = 'agent' AND \"resources\".\"status\" IN ('active', 'stale') AND \"resources\".\"repo_canonical_key\" IS NOT NULL", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "resources_organization_id_organizations_id_fk": { + "name": "resources_organization_id_organizations_id_fk", + "tableFrom": "resources", + "tableTo": "organizations", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "resources_owner_agent_id_agents_uuid_fk": { + "name": "resources_owner_agent_id_agents_uuid_fk", + "tableFrom": "resources", + "tableTo": "agents", + "columnsFrom": [ + "owner_agent_id" + ], + "columnsTo": [ + "uuid" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.server_instances": { + "name": "server_instances", + "schema": "", + "columns": { + "instance_id": { + "name": "instance_id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "last_heartbeat": { + "name": "last_heartbeat", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.session_events": { + "name": "session_events", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "chat_id": { + "name": "chat_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "seq": { + "name": "seq", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "payload": { + "name": "payload", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "uq_session_events_chat_seq": { + "name": "uq_session_events_chat_seq", + "columns": [ + { + "expression": "agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "seq", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_session_events_chat_created": { + "name": "idx_session_events_chat_created", + "columns": [ + { + "expression": "agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "chat_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": false, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_session_events_context_tree_usage_recent": { + "name": "idx_session_events_context_tree_usage_recent", + "columns": [ + { + "expression": "created_at", + "isExpression": false, + "asc": false, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"session_events\".\"kind\" = 'context_tree_usage'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_session_events_context_tree_io_agent_recent": { + "name": "idx_session_events_context_tree_io_agent_recent", + "columns": [ + { + "expression": "agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": false, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"session_events\".\"kind\" IN ('context_tree_usage', 'tool_call')", + "concurrently": false, + "method": "btree", + "with": {} + }, + "idx_session_events_token_usage_agent_recent": { + "name": "idx_session_events_token_usage_agent_recent", + "columns": [ + { + "expression": "agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": false, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"session_events\".\"kind\" = 'token_usage'", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.users": { + "name": "users", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "username": { + "name": "username", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "password_hash": { + "name": "password_hash", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "display_name": { + "name": "display_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "avatar_url": { + "name": "avatar_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'active'" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "users_username_unique": { + "name": "users_username_unique", + "nullsNotDistinct": false, + "columns": [ + "username" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + } + }, + "enums": {}, + "schemas": {}, + "sequences": {}, + "roles": {}, + "policies": {}, + "views": {}, + "_meta": { + "columns": {}, + "schemas": {}, + "tables": {} + } +} \ No newline at end of file diff --git a/packages/server/drizzle/meta/_journal.json b/packages/server/drizzle/meta/_journal.json index a36fd6a0e..6be50ec3c 100644 --- a/packages/server/drizzle/meta/_journal.json +++ b/packages/server/drizzle/meta/_journal.json @@ -582,6 +582,13 @@ "when": 1784778296618, "tag": "0082_heavy_moonstone", "breakpoints": true + }, + { + "idx": 83, + "version": "7", + "when": 1784803448712, + "tag": "0083_overconfident_maelstrom", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/server/package.json b/packages/server/package.json index a7ddccaf0..844a2261f 100644 --- a/packages/server/package.json +++ b/packages/server/package.json @@ -24,10 +24,13 @@ "coverage": "vitest run --coverage", "db:generate": "drizzle-kit generate && node ../../scripts/sync-migration-head.mjs", "db:migrate": "drizzle-kit migrate", + "migrate:attachments": "tsx --env-file-if-exists=../../.env scripts/migrate-attachments.ts", "db:studio": "drizzle-kit studio" }, "dependencies": { "@autotelic/fastify-opentelemetry": "^0.23.0", + "@aws-sdk/client-s3": "^3.1093.0", + "@aws-sdk/s3-request-presigner": "^3.1093.0", "@fastify/cors": "^11.2.0", "@fastify/rate-limit": "^10.3.0", "@fastify/static": "^9.0.0", diff --git a/packages/server/scripts/migrate-attachments.ts b/packages/server/scripts/migrate-attachments.ts new file mode 100644 index 000000000..d2b5b5059 --- /dev/null +++ b/packages/server/scripts/migrate-attachments.ts @@ -0,0 +1,47 @@ +/** + * Operator entry point for the attachment → object-storage data migration. + * All logic lives in src/services/attachment-migration.ts (tested there); + * this shell resolves config the same way the server does, refuses to run + * without object storage, and maps the verify phase onto the exit code. + * + * Run: pnpm --filter @first-tree/server migrate:attachments + * (requires FIRST_TREE_DATABASE_URL + FIRST_TREE_S3_* in the environment + * or ../../.env, exactly like the server itself) + */ + +import { createServerConfigSchema, initConfig } from "@first-tree/shared/config"; +import { connectDatabase } from "../src/db/connection.js"; +import { migrateAttachmentsToObjectStorage } from "../src/services/attachment-migration.js"; +import { createObjectStorage } from "../src/services/object-storage.js"; + +async function main(): Promise { + const config = await initConfig({ schema: createServerConfigSchema(), role: "server" }); + if (!config.objectStorage) { + console.error("Object storage is not configured (FIRST_TREE_S3_*); refusing to run."); + process.exit(1); + } + const db = connectDatabase(config.database.url); + const storage = createObjectStorage(config.objectStorage); + await storage.ensureBucket(); + + try { + const stats = await migrateAttachmentsToObjectStorage(db, storage); + if (stats.attachmentsRemaining > 0 || stats.avatarsRemaining > 0) { + console.error( + `Migration incomplete: ${stats.attachmentsRemaining} attachment / ${stats.avatarsRemaining} avatar payloads still inline. Rerun this command (idempotent) and investigate.`, + ); + process.exitCode = 1; + return; + } + console.log( + "Migration complete. Messages sent while the reference backfill ran may have added new references; rerunning is cheap and converges.", + ); + } finally { + await db.end(); + } +} + +main().catch((error) => { + console.error(error); + process.exit(1); +}); diff --git a/packages/server/src/__tests__/admin-agents.test.ts b/packages/server/src/__tests__/admin-agents.test.ts index c7375b45a..8b297eee5 100644 --- a/packages/server/src/__tests__/admin-agents.test.ts +++ b/packages/server/src/__tests__/admin-agents.test.ts @@ -7,10 +7,10 @@ import { organizations } from "../db/schema/organizations.js"; import { createAgent } from "../services/agent.js"; import { bindAgent, unbindAgent } from "../services/presence.js"; import { uuidv7 } from "../uuid.js"; -import { createAdminContext, createTestAdmin, useTestApp } from "./helpers.js"; +import { createAdminContext, createTestAdmin, useTestApp, workerObjectStorage } from "./helpers.js"; describe("Admin Agents API", () => { - const getApp = useTestApp(); + const getApp = useTestApp({ objectStorage: workerObjectStorage() }); async function authedRequest(app: FastifyInstance) { const ctx = await createAdminContext(app); @@ -411,6 +411,39 @@ describe("Admin Agents API", () => { expect(okRes.statusCode).toBe(204); }); + it("avatar serving stays proxied even in redirect download mode", async () => { + const { createTestApp, workerObjectStorage: workerStorage } = await import("./helpers.js"); + const app = await createTestApp({ + objectStorage: workerStorage(), + attachments: { downloadMode: "redirect" }, + }); + try { + const ctx = await createAdminContext(app); + const agent = await createAgent(app.db, { + name: `avatar-proxy-pin-${crypto.randomUUID().slice(0, 6)}`, + type: "agent", + managerId: ctx.memberId, + clientId: ctx.clientId, + }); + const bytes = Buffer.from("avatar-proxy-pin"); + const upload = await app.inject({ + method: "PUT", + url: `/api/v1/agents/${agent.uuid}/avatar`, + headers: { authorization: `Bearer ${ctx.accessToken}`, "content-type": "image/png" }, + payload: bytes, + }); + expect(upload.statusCode).toBe(200); + + // A presigned 302 would vary per request and defeat the immutable + // browser cache this surface depends on — avatars always proxy. + const serve = await app.inject({ method: "GET", url: `/api/v1/agents/${agent.uuid}/avatar` }); + expect(serve.statusCode).toBe(200); + expect(serve.rawPayload.equals(bytes)).toBe(true); + } finally { + await app.close(); + } + }); + it("serves uploaded agent avatars publicly and clears them through the manage route", async () => { const app = getApp(); const { ctx } = await authedRequest(app); @@ -437,6 +470,19 @@ describe("Admin Agents API", () => { expect(badUpload.statusCode).toBe(400); expect(badUpload.json<{ error: string }>().error).toContain("image/* Content-Type"); + // No body at all → no Content-Length → 411 (declared size is required + // since the payload streams to object storage). + const noLengthUpload = await app.inject({ + method: "PUT", + url: `/api/v1/agents/${agent.uuid}/avatar`, + headers: { + authorization: `Bearer ${ctx.accessToken}`, + "content-type": "image/png", + }, + }); + expect(noLengthUpload.statusCode).toBe(411); + expect(noLengthUpload.json<{ error: string }>().error).toContain("Content-Length"); + const emptyImageUpload = await app.inject({ method: "PUT", url: `/api/v1/agents/${agent.uuid}/avatar`, @@ -444,6 +490,7 @@ describe("Admin Agents API", () => { authorization: `Bearer ${ctx.accessToken}`, "content-type": "image/png", }, + payload: Buffer.alloc(0), }); expect(emptyImageUpload.statusCode).toBe(400); expect(emptyImageUpload.json<{ error: string }>().error).toContain("Avatar image payload is empty"); diff --git a/packages/server/src/__tests__/agent-service-extra.test.ts b/packages/server/src/__tests__/agent-service-extra.test.ts index 7da56b689..593fbdd69 100644 --- a/packages/server/src/__tests__/agent-service-extra.test.ts +++ b/packages/server/src/__tests__/agent-service-extra.test.ts @@ -1,3 +1,4 @@ +import { Readable } from "node:stream"; import { AGENT_STATUSES } from "@first-tree/shared"; import { eq } from "drizzle-orm"; import { describe, expect, it } from "vitest"; @@ -26,8 +27,9 @@ import { updateAgentSkills, } from "../services/agent.js"; import { createMember } from "../services/member.js"; +import { createObjectStorage } from "../services/object-storage.js"; import { createOrganization } from "../services/organization.js"; -import { createAdminContext, useTestApp } from "./helpers.js"; +import { createAdminContext, useTestApp, workerObjectStorage } from "./helpers.js"; describe("agent service extra coverage", () => { const getApp = useTestApp(); @@ -436,29 +438,36 @@ describe("agent service extra coverage", () => { clientId: admin.clientId, }); + const storage = createObjectStorage(workerObjectStorage()); + const avatarBody = (payload: Buffer) => Readable.from([payload]); + const setAvatar = (uuid: string, payload: Buffer, mime: string, contentLength = payload.byteLength) => + setAgentAvatarImage(app.db, storage, uuid, avatarBody(payload), { mime, contentLength }); + await expect(getAgentAvatarImage(app.db, agent.uuid)).resolves.toBeNull(); - await expect(setAgentAvatarImage(app.db, agent.uuid, Buffer.from("avatar"), "image/gif")).rejects.toBeInstanceOf( + await expect(setAvatar(agent.uuid, Buffer.from("avatar"), "image/gif")).rejects.toBeInstanceOf(BadRequestError); + await expect(setAvatar(agent.uuid, Buffer.alloc(0), "image/png")).rejects.toBeInstanceOf(BadRequestError); + await expect(setAvatar(agent.uuid, Buffer.alloc(MAX_AVATAR_IMAGE_BYTES + 1), "image/png")).rejects.toBeInstanceOf( BadRequestError, ); - await expect(setAgentAvatarImage(app.db, agent.uuid, Buffer.alloc(0), "image/png")).rejects.toBeInstanceOf( - BadRequestError, + await expect(setAvatar(crypto.randomUUID(), Buffer.from("avatar"), "image/png")).rejects.toBeInstanceOf( + NotFoundError, ); - await expect( - setAgentAvatarImage(app.db, agent.uuid, Buffer.alloc(MAX_AVATAR_IMAGE_BYTES + 1), "image/png"), - ).rejects.toBeInstanceOf(BadRequestError); - await expect( - setAgentAvatarImage(app.db, crypto.randomUUID(), Buffer.from("avatar"), "image/png"), - ).rejects.toBeInstanceOf(NotFoundError); + // Declared length is a contract: a mismatching body fails the stream. + await expect(setAvatar(agent.uuid, Buffer.from("avatar"), "image/png", 3)).rejects.toBeInstanceOf(BadRequestError); - const updatedAt = await setAgentAvatarImage(app.db, agent.uuid, Buffer.from("avatar"), "image/png"); + const updatedAt = await setAvatar(agent.uuid, Buffer.from("avatar"), "image/png"); await expect(getAgentAvatarImage(app.db, agent.uuid)).resolves.toEqual({ - data: Buffer.from("avatar"), + data: null, + objectKey: `avatars/${agent.uuid}`, mime: "image/png", updatedAt, }); + const stored = await storage.getObjectStream(`avatars/${agent.uuid}`); + expect(stored).not.toBeNull(); - await clearAgentAvatarImage(app.db, agent.uuid); + await clearAgentAvatarImage(app.db, storage, agent.uuid); await expect(getAgentAvatarImage(app.db, agent.uuid)).resolves.toBeNull(); - await expect(clearAgentAvatarImage(app.db, crypto.randomUUID())).rejects.toBeInstanceOf(NotFoundError); + await expect(storage.getObjectStream(`avatars/${agent.uuid}`)).resolves.toBeNull(); + await expect(clearAgentAvatarImage(app.db, storage, crypto.randomUUID())).rejects.toBeInstanceOf(NotFoundError); }); }); diff --git a/packages/server/src/__tests__/api-small-routes-extra.test.ts b/packages/server/src/__tests__/api-small-routes-extra.test.ts index 5ce17b5fc..a66f43f5b 100644 --- a/packages/server/src/__tests__/api-small-routes-extra.test.ts +++ b/packages/server/src/__tests__/api-small-routes-extra.test.ts @@ -683,6 +683,14 @@ describe("small API route handlers", () => { headers: { "content-type": "image/png" }, params: { uuid: "agent_1" }, }), + ).rejects.toThrow("Avatar uploads must declare Content-Length"); + + await expect( + route(routes, "PUT", "/:uuid/avatar").handler({ + body: { not: "bytes" }, + headers: { "content-type": "image/png", "content-length": "5" }, + params: { uuid: "agent_1" }, + }), ).rejects.toThrow("Avatar upload body must be raw image bytes."); }); diff --git a/packages/server/src/__tests__/attachment-migration.test.ts b/packages/server/src/__tests__/attachment-migration.test.ts new file mode 100644 index 000000000..32c2162eb --- /dev/null +++ b/packages/server/src/__tests__/attachment-migration.test.ts @@ -0,0 +1,276 @@ +import { Readable } from "node:stream"; +import { eq } from "drizzle-orm"; +import { describe, expect, it } from "vitest"; +import { agents } from "../db/schema/agents.js"; +import { attachmentReferences } from "../db/schema/attachment-references.js"; +import { attachments } from "../db/schema/attachments.js"; +import { messages } from "../db/schema/messages.js"; +import { createLegacyAttachment } from "../services/attachment.js"; +import { migrateAttachmentsToObjectStorage } from "../services/attachment-migration.js"; +import { createChat } from "../services/chat.js"; +import { attachmentObjectKey, avatarObjectKey, createObjectStorage } from "../services/object-storage.js"; +import { uuidv7 } from "../uuid.js"; +import { createTestAdmin, createTestAgent, useTestApp, workerObjectStorage } from "./helpers.js"; + +async function streamToBuffer(stream: NodeJS.ReadableStream): Promise { + const chunks: Buffer[] = []; + for await (const chunk of stream) { + chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)); + } + return Buffer.concat(chunks); +} + +describe("migrate:attachments — bytea to object storage", () => { + const getApp = useTestApp({ objectStorage: workerObjectStorage() }); + const storage = () => createObjectStorage(workerObjectStorage()); + + it("moves payloads, backfills org + edges, keeps downloads working, and is idempotent", async () => { + const app = getApp(); + const uid = crypto.randomUUID().slice(0, 6); + const uploader = await createTestAgent(app, { name: `mig-up-${uid}` }); + const { agent: peer } = await createTestAgent(app, { name: `mig-peer-${uid}`, type: "human" }); + + // Legacy attachments: payload inline, no org, no object key. + const payloadA = Buffer.from(`legacy-payload-A-${uid}`); + const payloadB = Buffer.from(`legacy-payload-B-${uid}`); + const legacyA = await createLegacyAttachment(app.db, { + mimeType: "image/png", + filename: "a.png", + data: payloadA, + uploadedBy: uploader.agent.uuid, + }); + const legacyB = await createLegacyAttachment(app.db, { + mimeType: "text/plain", + filename: "b.txt", + data: payloadB, + uploadedBy: uploader.agent.uuid, + }); + // An orphan uploader: no agent row backs this uploadedBy → org stays NULL. + const orphanUploader = await createLegacyAttachment(app.db, { + mimeType: "text/plain", + filename: "orphan-uploader.txt", + data: Buffer.from("x"), + uploadedBy: crypto.randomUUID(), + }); + + // Historic messages referencing legacyA (single) and legacyB (metadata), + // plus a dangling id that was never existence-checked — inserted + // directly, as the pre-ledger code produced them (no edges). + const chat = await createChat(app.db, uploader.agent.uuid, { type: "group", participantIds: [peer.uuid] }); + const danglingId = crypto.randomUUID(); + await app.db.insert(messages).values([ + { + id: uuidv7(), + chatId: chat.id, + senderId: uploader.agent.uuid, + format: "file", + content: { imageId: legacyA.id, mimeType: "image/png", filename: "a.png" }, + metadata: {}, + source: "api", + }, + { + id: uuidv7(), + chatId: chat.id, + senderId: uploader.agent.uuid, + format: "text", + content: "see attached doc", + metadata: { + attachments: [ + { + attachmentId: legacyB.id, + kind: "document", + mimeType: "text/plain", + filename: "b.txt", + size: payloadB.byteLength, + }, + ], + }, + source: "api", + }, + { + id: uuidv7(), + chatId: chat.id, + senderId: uploader.agent.uuid, + format: "file", + content: { imageId: danglingId, mimeType: "image/png", filename: "ghost.png" }, + metadata: {}, + source: "api", + }, + ]); + + // Legacy avatar: inline bytea on the agent row. + const avatarPayload = Buffer.from(`legacy-avatar-${uid}`); + await app.db + .update(agents) + .set({ avatarImageData: avatarPayload, avatarImageMime: "image/png", avatarImageUpdatedAt: new Date() }) + .where(eq(agents.uuid, uploader.agent.uuid)); + + const stats = await migrateAttachmentsToObjectStorage(app.db, storage()); + + // Payloads moved and byte-identical in object storage; rows hold keys only. + for (const [att, payload] of [ + [legacyA, payloadA], + [legacyB, payloadB], + ] as const) { + const [row] = await app.db.select().from(attachments).where(eq(attachments.id, att.id)); + expect(row?.objectKey).toBe(attachmentObjectKey(att.id)); + expect(row?.data).toBeNull(); + expect(row?.organizationId).toBe(uploader.agent.organizationId); + const object = await storage().getObjectStream(attachmentObjectKey(att.id)); + expect(object).not.toBeNull(); + if (object) { + expect((await streamToBuffer(object.body)).equals(payload)).toBe(true); + } + } + + // Orphan-uploader row: payload moved, org stays NULL (quota-exempt legacy). + const [orphanRow] = await app.db.select().from(attachments).where(eq(attachments.id, orphanUploader.id)); + expect(orphanRow?.organizationId).toBeNull(); + expect(orphanRow?.data).toBeNull(); + + // Edges: real references recorded, dangling id filtered. + const edgesA = await app.db + .select() + .from(attachmentReferences) + .where(eq(attachmentReferences.attachmentId, legacyA.id)); + expect(edgesA).toHaveLength(1); + const edgesB = await app.db + .select() + .from(attachmentReferences) + .where(eq(attachmentReferences.attachmentId, legacyB.id)); + expect(edgesB).toHaveLength(1); + expect(stats.edgesInserted).toBeGreaterThanOrEqual(2); + + // Avatar moved. + const [agentRow] = await app.db + .select({ objectKey: agents.avatarObjectKey, data: agents.avatarImageData }) + .from(agents) + .where(eq(agents.uuid, uploader.agent.uuid)); + expect(agentRow?.objectKey).toBe(avatarObjectKey(uploader.agent.uuid)); + expect(agentRow?.data).toBeNull(); + const avatarObject = await storage().getObjectStream(avatarObjectKey(uploader.agent.uuid)); + expect(avatarObject).not.toBeNull(); + if (avatarObject) { + expect((await streamToBuffer(avatarObject.body)).equals(avatarPayload)).toBe(true); + } + + // Verify phase reports completion... + expect(stats.attachmentsRemaining).toBe(0); + expect(stats.avatarsRemaining).toBe(0); + + // ...and pre-migration attachments still download through the API. + const admin = await createTestAdmin(app, { username: `mig-dl-${uid}` }); + const download = await app.inject({ + method: "GET", + url: `/api/v1/attachments/${legacyA.id}`, + headers: { authorization: `Bearer ${admin.accessToken}` }, + }); + expect(download.statusCode).toBe(200); + expect(download.rawPayload.equals(payloadA)).toBe(true); + + // Idempotent: a second run finds nothing to do and changes nothing. + const second = await migrateAttachmentsToObjectStorage(app.db, storage()); + expect(second.attachmentsMoved).toBe(0); + expect(second.avatarsMoved).toBe(0); + expect(second.edgesInserted).toBe(0); + expect(second.attachmentsRemaining).toBe(0); + }); + + it("0-row phase C swap keeps the object when a rival run already migrated the row", async () => { + const app = getApp(); + const uid = crypto.randomUUID().slice(0, 6); + const uploader = await createTestAgent(app, { name: `mig-rival-${uid}` }); + const payload = Buffer.from(`rival-${uid}`); + const legacy = await createLegacyAttachment(app.db, { + mimeType: "text/plain", + filename: "rival.txt", + data: payload, + uploadedBy: uploader.agent.uuid, + }); + + const stats = await migrateAttachmentsToObjectStorage(app.db, storage(), { + beforeAttachmentUpdate: async (attachmentId) => { + // A rival run wins the swap between our PUT and our UPDATE. + await app.db + .update(attachments) + .set({ objectKey: attachmentObjectKey(attachmentId), data: null }) + .where(eq(attachments.id, attachmentId)); + }, + }); + expect(stats.attachmentsSkipped).toBe(1); + + // The row owns the key — the object must have survived our 0-row branch. + const [row] = await app.db.select().from(attachments).where(eq(attachments.id, legacy.id)); + expect(row?.objectKey).toBe(attachmentObjectKey(legacy.id)); + const object = await storage().getObjectStream(attachmentObjectKey(legacy.id)); + expect(object).not.toBeNull(); + if (object) { + expect((await streamToBuffer(object.body)).equals(payload)).toBe(true); + } + }); + + it("0-row phase C swap deletes the object when the sweep destroyed the row mid-flight", async () => { + const app = getApp(); + const uid = crypto.randomUUID().slice(0, 6); + const uploader = await createTestAgent(app, { name: `mig-swept-${uid}` }); + const legacy = await createLegacyAttachment(app.db, { + mimeType: "text/plain", + filename: "swept.txt", + data: Buffer.from(`swept-${uid}`), + uploadedBy: uploader.agent.uuid, + }); + + const stats = await migrateAttachmentsToObjectStorage(app.db, storage(), { + beforeAttachmentUpdate: async (attachmentId) => { + // Sweep tombstoned + destroyed the row while our PUT was in flight. + await app.db.delete(attachments).where(eq(attachments.id, attachmentId)); + }, + }); + expect(stats.attachmentsSkipped).toBe(1); + + // Ownerless object must not leak (no row → the sweep can never see it). + await expect(storage().getObjectStream(attachmentObjectKey(legacy.id))).resolves.toBeNull(); + }); + + it("0-row phase D swap keeps the avatar a user uploaded mid-migration", async () => { + const app = getApp(); + const uid = crypto.randomUUID().slice(0, 6); + const { agent } = await createTestAgent(app, { name: `mig-av-race-${uid}` }); + await app.db + .update(agents) + .set({ + avatarImageData: Buffer.from("old-avatar"), + avatarImageMime: "image/png", + avatarImageUpdatedAt: new Date(), + }) + .where(eq(agents.uuid, agent.uuid)); + + const newAvatar = Buffer.from(`new-avatar-${uid}`); + const stats = await migrateAttachmentsToObjectStorage(app.db, storage(), { + beforeAvatarUpdate: async (agentUuid) => { + // An online avatar upload lands between our PUT and our UPDATE. + const { setAgentAvatarImage } = await import("../services/agent.js"); + await setAgentAvatarImage(app.db, storage(), agentUuid, Readable.from([newAvatar]), { + mime: "image/png", + contentLength: newAvatar.byteLength, + }); + }, + }); + expect(stats.avatarsSkipped).toBe(1); + expect(stats.avatarsMoved).toBe(0); + + // The freshly uploaded avatar must survive: row keeps its key and the + // object holds the NEW bytes (the online upload wrote after our PUT). + const [row] = await app.db + .select({ objectKey: agents.avatarObjectKey, data: agents.avatarImageData }) + .from(agents) + .where(eq(agents.uuid, agent.uuid)); + expect(row?.objectKey).toBe(avatarObjectKey(agent.uuid)); + expect(row?.data).toBeNull(); + const object = await storage().getObjectStream(avatarObjectKey(agent.uuid)); + expect(object).not.toBeNull(); + if (object) { + expect((await streamToBuffer(object.body)).equals(newAvatar)).toBe(true); + } + }); +}); diff --git a/packages/server/src/__tests__/attachment-quota.test.ts b/packages/server/src/__tests__/attachment-quota.test.ts new file mode 100644 index 000000000..3fb896bb8 --- /dev/null +++ b/packages/server/src/__tests__/attachment-quota.test.ts @@ -0,0 +1,166 @@ +import { PassThrough } from "node:stream"; +import { ATTACHMENT_ERROR_CODES, ATTACHMENT_FILENAME_HEADER, ATTACHMENT_MIME_HEADER } from "@first-tree/shared"; +import { and, eq, inArray, sql } from "drizzle-orm"; +import type { FastifyInstance } from "fastify"; +import { describe, expect, it } from "vitest"; +import { attachments } from "../db/schema/attachments.js"; +import { TooManyRequestsError } from "../errors.js"; +import { createUploadGate } from "../services/upload-gate.js"; +import { createTestAdmin, createTestApp, workerObjectStorage } from "./helpers.js"; + +type Admin = Awaited>; + +function upload(app: FastifyInstance, caller: Admin, payload: Buffer, filename = "q.bin") { + return app.inject({ + method: "POST", + url: `/api/v1/orgs/${caller.organizationId}/attachments`, + headers: { + authorization: `Bearer ${caller.accessToken}`, + "content-type": "application/octet-stream", + [ATTACHMENT_MIME_HEADER]: "application/octet-stream", + [ATTACHMENT_FILENAME_HEADER]: filename, + }, + payload, + }); +} + +describe("attachment org quotas — hard reject", () => { + it("rejects with 422 + stable code when the byte quota would be exceeded", async () => { + const app = await createTestApp({ + objectStorage: workerObjectStorage(), + attachments: { orgQuotaBytes: 100 }, + }); + try { + const admin = await createTestAdmin(app, { username: `qb-${crypto.randomUUID().slice(0, 6)}` }); + expect((await upload(app, admin, Buffer.alloc(60))).statusCode).toBe(201); + + const reply = await upload(app, admin, Buffer.alloc(60)); + expect(reply.statusCode).toBe(422); + const body = reply.json() as { code?: string; error?: string }; + expect(body.code).toBe(ATTACHMENT_ERROR_CODES.quotaExceeded); + expect(body.error).toMatch(/storage quota/); + + // Hard reject means no soft admission: usage stays at one object. + expect((await upload(app, admin, Buffer.alloc(40))).statusCode).toBe(201); + } finally { + await app.close(); + } + }); + + it("rejects with 422 + stable code when the object-count quota would be exceeded", async () => { + const app = await createTestApp({ + objectStorage: workerObjectStorage(), + attachments: { orgQuotaCount: 1 }, + }); + try { + const admin = await createTestAdmin(app, { username: `qc-${crypto.randomUUID().slice(0, 6)}` }); + expect((await upload(app, admin, Buffer.alloc(8))).statusCode).toBe(201); + + const reply = await upload(app, admin, Buffer.alloc(8)); + expect(reply.statusCode).toBe(422); + const body = reply.json() as { code?: string; error?: string }; + expect(body.code).toBe(ATTACHMENT_ERROR_CODES.quotaExceeded); + expect(body.error).toMatch(/count quota/); + } finally { + await app.close(); + } + }); + + it("admits exactly one of two concurrent uploads that each fit but jointly exceed the quota", async () => { + const app = await createTestApp({ + objectStorage: workerObjectStorage(), + attachments: { orgQuotaBytes: 100 }, + }); + try { + const admin = await createTestAdmin(app, { username: `qr-${crypto.randomUUID().slice(0, 6)}` }); + const [a, b] = await Promise.all([ + upload(app, admin, Buffer.alloc(70), "race-a.bin"), + upload(app, admin, Buffer.alloc(70), "race-b.bin"), + ]); + const statuses = [a.statusCode, b.statusCode].sort(); + // The per-org advisory xact lock serializes admission: never both. + expect(statuses).toEqual([201, 422]); + + // DB-level invariant: admitted usage never exceeds the quota. + const [usage] = await app.db + .select({ totalBytes: sql`COALESCE(SUM(${attachments.sizeBytes}), 0)` }) + .from(attachments) + .where( + and(eq(attachments.organizationId, admin.organizationId), inArray(attachments.state, ["pending", "stored"])), + ); + expect(Number(usage?.totalBytes ?? 0)).toBeLessThanOrEqual(100); + } finally { + await app.close(); + } + }); + + it("answers 429 + stable code over the wire while an uploader holds its only slot", async () => { + const app = await createTestApp({ + objectStorage: workerObjectStorage(), + attachments: { maxConcurrentUploadsPerUploader: 1 }, + }); + try { + const admin = await createTestAdmin(app, { username: `qg-${crypto.randomUUID().slice(0, 6)}` }); + + // First upload holds its slot: the payload stream stays open until we + // end it, so the gate stays occupied deterministically. + const held = new PassThrough(); + const payload = Buffer.from("held-upload-body"); + const firstReply = app.inject({ + method: "POST", + url: `/api/v1/orgs/${admin.organizationId}/attachments`, + headers: { + authorization: `Bearer ${admin.accessToken}`, + "content-type": "application/octet-stream", + "content-length": String(payload.byteLength), + [ATTACHMENT_MIME_HEADER]: "application/octet-stream", + [ATTACHMENT_FILENAME_HEADER]: "held.bin", + }, + payload: held, + }); + + // The slot is taken as soon as the first request reaches the gate; + // poll (bounded) until the second upload observes the 429. + let second: Awaited> | undefined; + for (let attempt = 0; attempt < 100; attempt++) { + second = await upload(app, admin, Buffer.alloc(4), `probe-${attempt}.bin`); + if (second.statusCode === 429) break; + await new Promise((resolve) => setTimeout(resolve, 10)); + } + expect(second?.statusCode).toBe(429); + expect((second?.json() as { code?: string }).code).toBe(ATTACHMENT_ERROR_CODES.concurrencyExceeded); + + // Release the held stream — the first upload completes and frees the slot. + held.end(payload); + expect((await firstReply).statusCode).toBe(201); + expect((await upload(app, admin, Buffer.alloc(4), "after.bin")).statusCode).toBe(201); + } finally { + await app.close(); + } + }); + + it("upload gate bounds per-uploader concurrency with 429 + stable code", () => { + const gate = createUploadGate(2); + const releaseA = gate.acquire("uploader-1"); + const releaseB = gate.acquire("uploader-1"); + // Third concurrent slot for the same uploader is refused... + try { + gate.acquire("uploader-1"); + expect.unreachable("expected TooManyRequestsError"); + } catch (error) { + expect(error).toBeInstanceOf(TooManyRequestsError); + expect((error as TooManyRequestsError).attrs?.code).toBe(ATTACHMENT_ERROR_CODES.concurrencyExceeded); + } + // ...while other uploaders are unaffected, and release frees the slot. + const releaseOther = gate.acquire("uploader-2"); + releaseA(); + const releaseC = gate.acquire("uploader-1"); + // Double-release is a no-op, not an underflow. + releaseA(); + releaseB(); + releaseC(); + releaseOther(); + const again = gate.acquire("uploader-1"); + again(); + }); +}); diff --git a/packages/server/src/__tests__/attachment-references.test.ts b/packages/server/src/__tests__/attachment-references.test.ts new file mode 100644 index 000000000..0928930e3 --- /dev/null +++ b/packages/server/src/__tests__/attachment-references.test.ts @@ -0,0 +1,309 @@ +import { Readable } from "node:stream"; +import { and, eq } from "drizzle-orm"; +import { describe, expect, it } from "vitest"; +import { attachmentReferences } from "../db/schema/attachment-references.js"; +import { attachments } from "../db/schema/attachments.js"; +import { + type AttachmentMeta, + createLegacyAttachment, + finalizeAttachment, + reserveAttachment, +} from "../services/attachment.js"; +import { collectAttachmentIds, destroyDeletingAttachments } from "../services/attachment-references.js"; +import { createChat } from "../services/chat.js"; +import { editMessage, sendMessage } from "../services/message.js"; +import { attachmentObjectKey, createObjectStorage } from "../services/object-storage.js"; +import { createOrganization } from "../services/organization.js"; +import { createTestAgent, useTestApp, workerObjectStorage } from "./helpers.js"; + +const QUOTA = { maxTotalBytes: 1024 * 1024, maxObjectCount: 100 }; + +describe("attachment reference lifecycle", () => { + const getApp = useTestApp({ objectStorage: workerObjectStorage() }); + const storage = () => createObjectStorage(workerObjectStorage()); + + async function setup(uid: string) { + const app = getApp(); + const sender = await createTestAgent(app, { name: `ref-sender-${uid}` }); + const { agent: peer } = await createTestAgent(app, { name: `ref-peer-${uid}`, type: "human" }); + const chat = await createChat(app.db, sender.agent.uuid, { type: "group", participantIds: [peer.uuid] }); + const organizationId = sender.agent.organizationId; + return { app, sender, peer, chat, organizationId }; + } + + /** Reserve + upload + finalize a real stored attachment owned by `orgId`. */ + async function storedAttachment( + app: ReturnType, + orgId: string, + uploadedBy: string, + payload = Buffer.from("attachment-bytes"), + ): Promise { + const reserved = await reserveAttachment(app.db, { + organizationId: orgId, + mimeType: "image/png", + filename: "ref.png", + sizeBytes: payload.byteLength, + uploadedBy, + quota: QUOTA, + }); + if (!reserved.objectKey) throw new Error("reservation missing object key"); + await storage().putObjectStream(reserved.objectKey, Readable.from([payload]), { + contentLength: payload.byteLength, + contentType: "image/png", + }); + expect(await finalizeAttachment(app.db, reserved.id)).toBe(true); + return reserved; + } + + function imageContent(att: AttachmentMeta) { + return { imageId: att.id, mimeType: "image/png" as const, filename: att.filename }; + } + + async function edgesOf(app: ReturnType, messageId: string): Promise { + const rows = await app.db + .select({ attachmentId: attachmentReferences.attachmentId }) + .from(attachmentReferences) + .where(eq(attachmentReferences.messageId, messageId)); + return rows.map((r) => r.attachmentId).sort(); + } + + it("collectAttachmentIds covers single, batch, and metadata shapes (deduped)", () => { + const a = "11111111-1111-4111-8111-111111111111"; + const b = "22222222-2222-4222-8222-222222222222"; + expect(collectAttachmentIds({ imageId: a, mimeType: "image/png", filename: "x.png" }, {})).toEqual(new Set([a])); + expect( + collectAttachmentIds( + { + attachments: [ + { imageId: a, mimeType: "image/png", filename: "x.png" }, + { imageId: b, mimeType: "image/png", filename: "y.png" }, + ], + }, + undefined, + ), + ).toEqual(new Set([a, b])); + expect( + collectAttachmentIds("plain text", { + attachments: [{ attachmentId: a, kind: "document", mimeType: "text/plain", filename: "d.txt", size: 3 }], + }), + ).toEqual(new Set([a])); + // Same id in content and metadata collapses to one entry. + expect( + collectAttachmentIds( + { imageId: a, mimeType: "image/png", filename: "x.png" }, + { + attachments: [{ attachmentId: a, kind: "document", mimeType: "image/png", filename: "x.png", size: 3 }], + }, + ), + ).toEqual(new Set([a])); + expect(collectAttachmentIds("hello", {})).toEqual(new Set()); + }); + + it("send records edges for single-image content", async () => { + const uid = crypto.randomUUID().slice(0, 6); + const { app, sender, peer, chat, organizationId } = await setup(uid); + const att = await storedAttachment(app, organizationId, sender.agent.uuid); + + const { message } = await sendMessage(app.db, chat.id, sender.agent.uuid, { + source: "api", + format: "file", + content: imageContent(att), + metadata: { mentions: [peer.uuid] }, + }); + expect(await edgesOf(app, message.id)).toEqual([att.id]); + }); + + it("send records edges for batch content and metadata refs together", async () => { + const uid = crypto.randomUUID().slice(0, 6); + const { app, sender, peer, chat, organizationId } = await setup(uid); + const a = await storedAttachment(app, organizationId, sender.agent.uuid); + const b = await storedAttachment(app, organizationId, sender.agent.uuid); + const doc = await storedAttachment(app, organizationId, sender.agent.uuid, Buffer.from("doc-bytes")); + + const { message } = await sendMessage(app.db, chat.id, sender.agent.uuid, { + source: "api", + format: "file", + content: { caption: "two shots", attachments: [imageContent(a), imageContent(b)] }, + metadata: { + mentions: [peer.uuid], + attachments: [ + { + attachmentId: doc.id, + kind: "document", + mimeType: doc.mimeType, + filename: doc.filename, + size: doc.sizeBytes, + }, + ], + }, + }); + expect(await edgesOf(app, message.id)).toEqual([a.id, b.id, doc.id].sort()); + }); + + it("send rejects unknown, pending, deleting, and cross-org attachment references", async () => { + const uid = crypto.randomUUID().slice(0, 6); + const { app, sender, peer, chat, organizationId } = await setup(uid); + + const sendWith = (content: unknown) => + sendMessage(app.db, chat.id, sender.agent.uuid, { + source: "api", + format: "file", + content, + metadata: { mentions: [peer.uuid] }, + }); + + // Unknown id — shape-valid, existence-invalid. + await expect( + sendWith({ imageId: crypto.randomUUID(), mimeType: "image/png", filename: "ghost.png" }), + ).rejects.toThrow(/unknown attachment/); + + // Pending reservation (upload not finalized). + const pending = await reserveAttachment(app.db, { + organizationId, + mimeType: "image/png", + filename: "pending.png", + sizeBytes: 8, + uploadedBy: sender.agent.uuid, + quota: QUOTA, + }); + await expect(sendWith(imageContent(pending))).rejects.toThrow(/not available/); + + // Deleting tombstone. + const doomed = await storedAttachment(app, organizationId, sender.agent.uuid); + await app.db.update(attachments).set({ state: "deleting" }).where(eq(attachments.id, doomed.id)); + await expect(sendWith(imageContent(doomed))).rejects.toThrow(/not available/); + + // Cross-org reference. + const otherOrg = await createOrganization(app.db, { + name: `ref-org-${uid}`, + displayName: "Ref Other Org", + }); + const foreign = await storedAttachment(app, otherOrg.id, sender.agent.uuid); + await expect(sendWith(imageContent(foreign))).rejects.toThrow(/different organization/); + + // Legacy NULL-org rows stay referenceable (pre-backfill grandfathering). + const legacy = await createLegacyAttachment(app.db, { + mimeType: "image/png", + filename: "legacy.png", + data: Buffer.from("legacy"), + uploadedBy: sender.agent.uuid, + }); + const { message } = await sendWith(imageContent(legacy)); + expect(await edgesOf(app, message.id)).toEqual([legacy.id]); + }); + + it("edit dropping the last reference destroys the attachment (object + row)", async () => { + const uid = crypto.randomUUID().slice(0, 6); + const { app, sender, peer, chat, organizationId } = await setup(uid); + const att = await storedAttachment(app, organizationId, sender.agent.uuid); + + const { message } = await sendMessage(app.db, chat.id, sender.agent.uuid, { + source: "api", + format: "file", + content: imageContent(att), + metadata: { mentions: [peer.uuid] }, + }); + + await editMessage(app.db, storage(), chat.id, message.id, sender.agent.uuid, { + format: "text", + content: "image retracted", + }); + + expect(await edgesOf(app, message.id)).toEqual([]); + const [row] = await app.db.select().from(attachments).where(eq(attachments.id, att.id)); + expect(row).toBeUndefined(); + await expect(storage().getObjectStream(attachmentObjectKey(att.id))).resolves.toBeNull(); + }); + + it("edit replacing image A with image B moves the edge and destroys only A", async () => { + const uid = crypto.randomUUID().slice(0, 6); + const { app, sender, peer, chat, organizationId } = await setup(uid); + const a = await storedAttachment(app, organizationId, sender.agent.uuid); + const b = await storedAttachment(app, organizationId, sender.agent.uuid); + + const { message } = await sendMessage(app.db, chat.id, sender.agent.uuid, { + source: "api", + format: "file", + content: imageContent(a), + metadata: { mentions: [peer.uuid] }, + }); + + await editMessage(app.db, storage(), chat.id, message.id, sender.agent.uuid, { content: imageContent(b) }); + + expect(await edgesOf(app, message.id)).toEqual([b.id]); + const [rowA] = await app.db.select().from(attachments).where(eq(attachments.id, a.id)); + expect(rowA).toBeUndefined(); + const [rowB] = await app.db.select().from(attachments).where(eq(attachments.id, b.id)); + expect(rowB?.state).toBe("stored"); + }); + + it("an attachment still referenced by another message survives an edit", async () => { + const uid = crypto.randomUUID().slice(0, 6); + const { app, sender, peer, chat, organizationId } = await setup(uid); + const shared = await storedAttachment(app, organizationId, sender.agent.uuid); + + const first = await sendMessage(app.db, chat.id, sender.agent.uuid, { + source: "api", + format: "file", + content: imageContent(shared), + metadata: { mentions: [peer.uuid] }, + }); + const second = await sendMessage(app.db, chat.id, sender.agent.uuid, { + source: "api", + format: "file", + content: imageContent(shared), + metadata: { mentions: [peer.uuid] }, + }); + + await editMessage(app.db, storage(), chat.id, first.message.id, sender.agent.uuid, { + format: "text", + content: "retracted", + }); + + expect(await edgesOf(app, first.message.id)).toEqual([]); + expect(await edgesOf(app, second.message.id)).toEqual([shared.id]); + const [row] = await app.db.select().from(attachments).where(eq(attachments.id, shared.id)); + expect(row?.state).toBe("stored"); + expect(await storage().getObjectStream(attachmentObjectKey(shared.id))).not.toBeNull(); + }); + + it("an invalid content imageId surfaces as a wire 400 on the chat message route", async () => { + const uid = crypto.randomUUID().slice(0, 6); + const app = getApp(); + const { createTestAdmin } = await import("./helpers.js"); + const admin = await createTestAdmin(app, { username: `ref-wire-${uid}` }); + const { agent: peer } = await createTestAgent(app, { name: `ref-wire-peer-${uid}` }); + const chat = await createChat(app.db, admin.humanAgentUuid, { type: "group", participantIds: [peer.uuid] }); + + const reply = await app.inject({ + method: "POST", + url: `/api/v1/chats/${chat.id}/messages`, + headers: { authorization: `Bearer ${admin.accessToken}` }, + payload: { + format: "file", + content: { imageId: crypto.randomUUID(), mimeType: "image/png", filename: "ghost.png" }, + metadata: { mentions: [peer.uuid] }, + }, + }); + expect(reply.statusCode).toBe(400); + expect((reply.json() as { error: string }).error).toMatch(/unknown attachment/); + }); + + it("destroyDeletingAttachments leaves S3-backed tombstones alone when storage is unavailable", async () => { + const uid = crypto.randomUUID().slice(0, 6); + const { app, sender, organizationId } = await setup(uid); + const att = await storedAttachment(app, organizationId, sender.agent.uuid); + await app.db.update(attachments).set({ state: "deleting" }).where(eq(attachments.id, att.id)); + + await destroyDeletingAttachments(app.db, null, [att.id]); + const [kept] = await app.db + .select() + .from(attachments) + .where(and(eq(attachments.id, att.id), eq(attachments.state, "deleting"))); + expect(kept).toBeDefined(); + + await destroyDeletingAttachments(app.db, storage(), [att.id]); + const [gone] = await app.db.select().from(attachments).where(eq(attachments.id, att.id)); + expect(gone).toBeUndefined(); + }); +}); diff --git a/packages/server/src/__tests__/attachment-sweep.test.ts b/packages/server/src/__tests__/attachment-sweep.test.ts new file mode 100644 index 000000000..a300d634b --- /dev/null +++ b/packages/server/src/__tests__/attachment-sweep.test.ts @@ -0,0 +1,187 @@ +import { Readable } from "node:stream"; +import { eq } from "drizzle-orm"; +import { describe, expect, it } from "vitest"; +import { attachmentReferences } from "../db/schema/attachment-references.js"; +import { attachments } from "../db/schema/attachments.js"; +import { messages } from "../db/schema/messages.js"; +import { type AttachmentMeta, finalizeAttachment, reserveAttachment } from "../services/attachment.js"; +import { sweepAttachments } from "../services/attachment-sweep.js"; +import { createChat } from "../services/chat.js"; +import { attachmentObjectKey, createObjectStorage } from "../services/object-storage.js"; +import { uuidv7 } from "../uuid.js"; +import { createTestAgent, useTestApp, workerObjectStorage } from "./helpers.js"; + +const QUOTA = { maxTotalBytes: 1024 * 1024, maxObjectCount: 100 }; +const SWEEP_OPTS = { orphanGraceSeconds: 3600, pendingTtlSeconds: 600 }; + +describe("attachment orphan sweep", () => { + const getApp = useTestApp({ objectStorage: workerObjectStorage() }); + const storage = () => createObjectStorage(workerObjectStorage()); + + async function backdate(app: ReturnType, id: string, seconds: number): Promise { + await app.db + .update(attachments) + .set({ createdAt: new Date(Date.now() - seconds * 1000) }) + .where(eq(attachments.id, id)); + } + + async function reservedAttachment(app: ReturnType, uploadedBy: string): Promise { + const org = (await import("../services/organization.js")).resolveDefaultOrgId; + return reserveAttachment(app.db, { + organizationId: await org(app.db), + mimeType: "image/png", + filename: "sweep.png", + sizeBytes: 10, + uploadedBy, + quota: QUOTA, + }); + } + + async function storedAttachment(app: ReturnType, uploadedBy: string): Promise { + const reserved = await reservedAttachment(app, uploadedBy); + if (!reserved.objectKey) throw new Error("missing object key"); + const payload = Buffer.alloc(10, 1); + await storage().putObjectStream(reserved.objectKey, Readable.from([payload]), { + contentLength: payload.byteLength, + contentType: "image/png", + }); + await finalizeAttachment(app.db, reserved.id); + return reserved; + } + + async function rowOf(app: ReturnType, id: string) { + const [row] = await app.db.select().from(attachments).where(eq(attachments.id, id)); + return row; + } + + it("reclaims expired pending reservations and leaves fresh ones", async () => { + const app = getApp(); + const { agent } = await createTestAgent(app, { name: `sw-pend-${crypto.randomUUID().slice(0, 6)}` }); + const expired = await reservedAttachment(app, agent.uuid); + const fresh = await reservedAttachment(app, agent.uuid); + await backdate(app, expired.id, SWEEP_OPTS.pendingTtlSeconds + 60); + + const stats = await sweepAttachments(app.db, storage(), SWEEP_OPTS); + expect(stats.pendingReclaimed).toBe(1); + expect(await rowOf(app, expired.id)).toBeUndefined(); + expect((await rowOf(app, fresh.id))?.state).toBe("pending"); + }); + + it("deletes aged zero-edge stored attachments (object + row), keeps young and referenced ones", async () => { + const app = getApp(); + const { agent } = await createTestAgent(app, { name: `sw-orph-${crypto.randomUUID().slice(0, 6)}` }); + const { agent: peer } = await createTestAgent(app, { + name: `sw-peer-${crypto.randomUUID().slice(0, 6)}`, + type: "human", + }); + + const orphan = await storedAttachment(app, agent.uuid); + const young = await storedAttachment(app, agent.uuid); + const referenced = await storedAttachment(app, agent.uuid); + await backdate(app, orphan.id, SWEEP_OPTS.orphanGraceSeconds + 60); + await backdate(app, referenced.id, SWEEP_OPTS.orphanGraceSeconds + 60); + + // Give `referenced` a real ledger edge via an actual message. + const chat = await createChat(app.db, agent.uuid, { type: "group", participantIds: [peer.uuid] }); + const { sendMessage } = await import("../services/message.js"); + await sendMessage(app.db, chat.id, agent.uuid, { + source: "api", + format: "file", + content: { imageId: referenced.id, mimeType: "image/png", filename: referenced.filename }, + metadata: { mentions: [peer.uuid] }, + }); + + const stats = await sweepAttachments(app.db, storage(), SWEEP_OPTS); + expect(stats.orphansDeleted).toBe(1); + expect(await rowOf(app, orphan.id)).toBeUndefined(); + await expect(storage().getObjectStream(attachmentObjectKey(orphan.id))).resolves.toBeNull(); + expect((await rowOf(app, young.id))?.state).toBe("stored"); + expect((await rowOf(app, referenced.id))?.state).toBe("stored"); + }); + + it("verify scan vetoes candidates whose id appears in message text without a ledger edge", async () => { + const app = getApp(); + const { agent } = await createTestAgent(app, { name: `sw-veto-${crypto.randomUUID().slice(0, 6)}` }); + const { agent: peer } = await createTestAgent(app, { + name: `sw-vpeer-${crypto.randomUUID().slice(0, 6)}`, + type: "human", + }); + const preBackfill = await storedAttachment(app, agent.uuid); + await backdate(app, preBackfill.id, SWEEP_OPTS.orphanGraceSeconds + 60); + + // Simulate the deploy-before-backfill window: a message row references + // the id in content jsonb, but no `attachment_references` edge exists + // (written directly, bypassing the send path that would record one). + const chat = await createChat(app.db, agent.uuid, { type: "group", participantIds: [peer.uuid] }); + await app.db.insert(messages).values({ + id: uuidv7(), + chatId: chat.id, + senderId: agent.uuid, + format: "file", + content: { imageId: preBackfill.id, mimeType: "image/png", filename: "old.png" }, + metadata: {}, + source: "api", + }); + const edges = await app.db + .select() + .from(attachmentReferences) + .where(eq(attachmentReferences.attachmentId, preBackfill.id)); + expect(edges).toHaveLength(0); + + const stats = await sweepAttachments(app.db, storage(), SWEEP_OPTS); + expect(stats.orphansVetoed).toBe(1); + expect(stats.orphansDeleted).toBe(0); + expect((await rowOf(app, preBackfill.id))?.state).toBe("stored"); + await expect(storage().getObjectStream(attachmentObjectKey(preBackfill.id))).resolves.not.toBeNull(); + }); + + it("clears leftover deleting tombstones, including legacy rows without objects", async () => { + const app = getApp(); + const { agent } = await createTestAgent(app, { name: `sw-tomb-${crypto.randomUUID().slice(0, 6)}` }); + const tombstoned = await storedAttachment(app, agent.uuid); + await app.db.update(attachments).set({ state: "deleting" }).where(eq(attachments.id, tombstoned.id)); + + // Legacy shape: bytea payload, no object key, tombstoned. + const legacyId = crypto.randomUUID(); + await app.db.insert(attachments).values({ + id: legacyId, + organizationId: null, + mimeType: "text/plain", + filename: "legacy.txt", + sizeBytes: 6, + objectKey: null, + state: "deleting", + data: Buffer.from("legacy"), + uploadedBy: agent.uuid, + }); + + const stats = await sweepAttachments(app.db, storage(), SWEEP_OPTS); + expect(stats.tombstonesCleared).toBe(2); + expect(await rowOf(app, tombstoned.id)).toBeUndefined(); + expect(await rowOf(app, legacyId)).toBeUndefined(); + await expect(storage().getObjectStream(attachmentObjectKey(tombstoned.id))).resolves.toBeNull(); + }); + + it("concurrent sweeps split the work without double-processing errors", async () => { + const app = getApp(); + const { agent } = await createTestAgent(app, { name: `sw-conc-${crypto.randomUUID().slice(0, 6)}` }); + const ids: string[] = []; + for (let i = 0; i < 6; i++) { + const att = await storedAttachment(app, agent.uuid); + await backdate(app, att.id, SWEEP_OPTS.orphanGraceSeconds + 60); + ids.push(att.id); + } + + const [a, b] = await Promise.all([ + sweepAttachments(app.db, storage(), SWEEP_OPTS), + sweepAttachments(app.db, storage(), SWEEP_OPTS), + ]); + // Between the two runs every orphan is gone exactly once; SKIP LOCKED + // partitions the claims, and idempotent destroy tolerates overlap on + // the tombstone pass. + expect(a.orphansDeleted + b.orphansDeleted).toBeGreaterThanOrEqual(ids.length); + for (const id of ids) { + expect(await rowOf(app, id)).toBeUndefined(); + } + }); +}); diff --git a/packages/server/src/__tests__/attachments-route.test.ts b/packages/server/src/__tests__/attachments-route.test.ts index 51e6e7a21..f9e884f66 100644 --- a/packages/server/src/__tests__/attachments-route.test.ts +++ b/packages/server/src/__tests__/attachments-route.test.ts @@ -1,11 +1,48 @@ -import { ATTACHMENT_FILENAME_HEADER, ATTACHMENT_MIME_HEADER, MAX_ATTACHMENT_BYTES } from "@first-tree/shared"; +import { Readable } from "node:stream"; +import { ListObjectsV2Command, S3Client } from "@aws-sdk/client-s3"; +import { + ATTACHMENT_ERROR_CODES, + ATTACHMENT_FILENAME_HEADER, + ATTACHMENT_MIME_HEADER, + MAX_ATTACHMENT_BYTES, + ORG_ATTACHMENT_QUOTA_BYTES, + ORG_ATTACHMENT_QUOTA_COUNT, +} from "@first-tree/shared"; +import { eq } from "drizzle-orm"; import type { FastifyInstance } from "fastify"; import { describe, expect, it } from "vitest"; +import { attachments } from "../db/schema/attachments.js"; import { organizations } from "../db/schema/organizations.js"; -import { createAttachment } from "../services/attachment.js"; +import { createLegacyAttachment, reserveAttachment } from "../services/attachment.js"; import { ensureMembership } from "../services/membership.js"; +import { attachmentObjectKey, createObjectStorage } from "../services/object-storage.js"; import { uuidv7 } from "../uuid.js"; -import { createAdminContext, createTestAdmin, useTestApp } from "./helpers.js"; +import { createAdminContext, createTestAdmin, createTestApp, useTestApp, workerObjectStorage } from "./helpers.js"; + +const DEFAULT_QUOTA = { maxTotalBytes: ORG_ATTACHMENT_QUOTA_BYTES, maxObjectCount: ORG_ATTACHMENT_QUOTA_COUNT }; + +/** Total objects in this worker's bucket (files in a worker run sequentially). */ +async function countBucketObjects(): Promise { + const target = workerObjectStorage(); + const client = new S3Client({ + region: target.region, + endpoint: target.endpoint, + forcePathStyle: true, + credentials: { accessKeyId: target.accessKeyId, secretAccessKey: target.secretAccessKey }, + }); + try { + let count = 0; + let token: string | undefined; + do { + const page = await client.send(new ListObjectsV2Command({ Bucket: target.bucket, ContinuationToken: token })); + count += page.KeyCount ?? 0; + token = page.IsTruncated ? page.NextContinuationToken : undefined; + } while (token); + return count; + } finally { + client.destroy(); + } +} type Admin = Awaited>; @@ -38,7 +75,7 @@ function getAttachment(app: FastifyInstance, caller: Admin, id: string) { } describe("attachments route — upload + capability download", () => { - const getApp = useTestApp(); + const getApp = useTestApp({ objectStorage: workerObjectStorage() }); it("uploads then downloads via uploader", async () => { const app = getApp(); @@ -52,6 +89,16 @@ describe("attachments route — upload + capability download", () => { expect(body.sizeBytes).toBe(bytes.byteLength); expect(body.uploadedBy).toBe(admin.humanAgentUuid); + // The payload landed in object storage — the PG row holds metadata only. + const [row] = await app.db.select().from(attachments).where(eq(attachments.id, body.id)); + expect(row?.state).toBe("stored"); + expect(row?.organizationId).toBe(admin.organizationId); + expect(row?.objectKey).toBe(attachmentObjectKey(body.id)); + expect(row?.data).toBeNull(); + const storage = createObjectStorage(workerObjectStorage()); + const object = await storage.getObjectStream(attachmentObjectKey(body.id)); + expect(object).not.toBeNull(); + const download = await getAttachment(app, admin, body.id); expect(download.statusCode).toBe(200); expect(download.headers["content-type"]).toBe("image/png"); @@ -59,7 +106,7 @@ describe("attachments route — upload + capability download", () => { expect(download.headers["x-content-type-options"]).toBe("nosniff"); expect(download.headers["cache-control"]).toBe("private, max-age=31536000, immutable"); expect(download.headers.etag).toBe(`"${body.id}"`); - expect(download.headers["content-disposition"]).toBe('inline; filename="kitten.png"'); + expect(download.headers["content-disposition"]).toBe(`inline; filename="kitten.png"; filename*=UTF-8''kitten.png`); expect(download.rawPayload.equals(bytes)).toBe(true); }); @@ -145,16 +192,18 @@ describe("attachments route — upload + capability download", () => { expect(blankMime.statusCode).toBe(400); await expect( - createAttachment(app.db, { + reserveAttachment(app.db, { + organizationId: admin.organizationId, mimeType: "image/png", filename: " ", - data: Buffer.from("filename"), + sizeBytes: 8, uploadedBy: admin.humanAgentUuid, + quota: DEFAULT_QUOTA, }), ).rejects.toThrow("Attachment filename is required"); }); - it("surfaces an empty insert-returning result from the attachment store", async () => { + it("surfaces an empty insert-returning result from the legacy attachment writer", async () => { const fakeDb = { insert: () => ({ values: () => ({ @@ -164,7 +213,7 @@ describe("attachments route — upload + capability download", () => { }; await expect( - createAttachment(fakeDb as never, { + createLegacyAttachment(fakeDb as never, { mimeType: "image/png", filename: "x.png", data: Buffer.from("bytes"), @@ -173,14 +222,116 @@ describe("attachments route — upload + capability download", () => { ).rejects.toThrow("Attachment insert returned no row"); }); - it("rejects oversize at bodyLimit (413) or service cap (400)", async () => { + it("rejects oversize uploads with 413 + stable code before reading the body", async () => { const app = getApp(); const admin = await createTestAdmin(app, { username: `os-${crypto.randomUUID().slice(0, 6)}` }); - // 1 KB over the cap — well under the route bodyLimit, so the service- - // layer cap is what fires. const oversize = Buffer.alloc(MAX_ATTACHMENT_BYTES + 1024); const reply = await postAttachment(app, admin, oversize); - expect([400, 413]).toContain(reply.statusCode); + expect(reply.statusCode).toBe(413); + expect((reply.json() as { code?: string }).code).toBe(ATTACHMENT_ERROR_CODES.tooLarge); + }); + + it("rejects uploads without Content-Length (chunked) with 411", async () => { + const app = getApp(); + const admin = await createTestAdmin(app, { username: `cl-${crypto.randomUUID().slice(0, 6)}` }); + const reply = await app.inject({ + method: "POST", + url: `/api/v1/orgs/${admin.organizationId}/attachments`, + headers: { + authorization: `Bearer ${admin.accessToken}`, + "content-type": "application/octet-stream", + [ATTACHMENT_MIME_HEADER]: "image/png", + [ATTACHMENT_FILENAME_HEADER]: "x.bin", + }, + // A stream payload makes inject send chunked transfer encoding. + payload: Readable.from([Buffer.from("hi")]), + }); + expect(reply.statusCode).toBe(411); + expect((reply.json() as { code?: string }).code).toBe(ATTACHMENT_ERROR_CODES.lengthRequired); + }); + + it("cleans up the reservation and object when the body undershoots Content-Length", async () => { + const app = getApp(); + const admin = await createTestAdmin(app, { username: `abrt-${crypto.randomUUID().slice(0, 6)}` }); + + const objectsBefore = await countBucketObjects(); + + // Declared 64 bytes, deliver 10, end normally — the byte-limit stream + // fails the upload at EOF (same cleanup path as a client abort). + const reply = await app.inject({ + method: "POST", + url: `/api/v1/orgs/${admin.organizationId}/attachments`, + headers: { + authorization: `Bearer ${admin.accessToken}`, + "content-type": "application/octet-stream", + "content-length": "64", + [ATTACHMENT_MIME_HEADER]: "application/octet-stream", + [ATTACHMENT_FILENAME_HEADER]: "truncated.bin", + }, + payload: Readable.from([Buffer.alloc(10)]), + }); + expect(reply.statusCode).toBe(400); + expect((reply.json() as { error: string }).error).toMatch(/does not match Content-Length/); + + // No reservation survives for this uploader... + const rows = await app.db.select().from(attachments).where(eq(attachments.uploadedBy, admin.humanAgentUuid)); + expect(rows).toHaveLength(0); + // ...and no object leaked into the bucket (files in one worker run + // sequentially, so the bucket count is stable across this test). + expect(await countBucketObjects()).toBe(objectsBefore); + }); + + it("redirect mode answers 302 with a working short-lived presigned URL", async () => { + const app = await createTestApp({ + objectStorage: workerObjectStorage(), + attachments: { downloadMode: "redirect" }, + }); + try { + const admin = await createTestAdmin(app, { username: `rd-${crypto.randomUUID().slice(0, 6)}` }); + const bytes = Buffer.from("redirect-me"); + const uploadReply = await postAttachment(app, admin, bytes, { filename: "r.bin", mime: "text/plain" }); + expect(uploadReply.statusCode).toBe(201); + const id = (uploadReply.json() as { id: string }).id; + + const reply = await getAttachment(app, admin, id); + expect(reply.statusCode).toBe(302); + expect(reply.headers["cache-control"]).toBe("private, no-store"); + const location = reply.headers.location; + expect(typeof location).toBe("string"); + expect(String(location)).toContain("X-Amz-Signature"); + + // The presigned URL works without any Authorization header and pins + // the response content headers. + const fetched = await fetch(String(location)); + expect(fetched.status).toBe(200); + expect(Buffer.from(await fetched.arrayBuffer()).equals(bytes)).toBe(true); + expect(fetched.headers.get("content-type")).toBe("text/plain"); + expect(fetched.headers.get("content-disposition")).toContain("inline"); + } finally { + await app.close(); + } + }); + + it("answers 503 when object storage is not configured", async () => { + const app = await createTestApp(); + try { + const admin = await createTestAdmin(app, { username: `nos3-${crypto.randomUUID().slice(0, 6)}` }); + const reply = await postAttachment(app, admin, Buffer.from("hi")); + expect(reply.statusCode).toBe(503); + + // Legacy bytea rows still download without object storage. + const legacy = await createLegacyAttachment(app.db, { + mimeType: "text/plain", + filename: "legacy.txt", + data: Buffer.from("pre-migration"), + uploadedBy: admin.humanAgentUuid, + }); + const download = await getAttachment(app, admin, legacy.id); + expect(download.statusCode).toBe(200); + expect(download.rawPayload.toString()).toBe("pre-migration"); + } finally { + await app.close(); + } }); it("returns 404 for unknown attachment id", async () => { diff --git a/packages/server/src/__tests__/background-tasks-extra.test.ts b/packages/server/src/__tests__/background-tasks-extra.test.ts index 728f38f1e..5d0c9833f 100644 --- a/packages/server/src/__tests__/background-tasks-extra.test.ts +++ b/packages/server/src/__tests__/background-tasks-extra.test.ts @@ -55,6 +55,13 @@ function makeApp(archiveSweepIntervalSeconds = 30): FastifyInstance { cronJobs: { enabled: false, }, + attachments: { + // Disabled: these unit tests pin the pre-existing timers; the + // attachment sweep has its own integration suite. + sweepIntervalSeconds: 0, + orphanGraceSeconds: 86_400, + pendingTtlSeconds: 3_600, + }, runtime: { archiveMappedIdleSeconds: 3_600, archiveSweepIntervalSeconds, @@ -63,6 +70,7 @@ function makeApp(archiveSweepIntervalSeconds = 30): FastifyInstance { }, }, db: { name: "db" }, + objectStorage: null, } as unknown as FastifyInstance; } diff --git a/packages/server/src/__tests__/bootstrap.test.ts b/packages/server/src/__tests__/bootstrap.test.ts index efa0a9c6f..77de03619 100644 --- a/packages/server/src/__tests__/bootstrap.test.ts +++ b/packages/server/src/__tests__/bootstrap.test.ts @@ -41,6 +41,15 @@ const baseServerConfig: ServerConfig = { docs: { enabled: false }, cronJobs: { enabled: false }, database: { url: process.env.DATABASE_URL ?? "", provider: "external" }, + attachments: { + downloadMode: "proxy", + orgQuotaBytes: 2 * 1024 * 1024 * 1024, + orgQuotaCount: 1000, + sweepIntervalSeconds: 0, + orphanGraceSeconds: 86_400, + pendingTtlSeconds: 3600, + maxConcurrentUploadsPerUploader: 4, + }, server: { port: 0, host: "127.0.0.1", publicUrl: "https://first-tree.example" }, workspace: { root: "/tmp/first-tree-test-workspaces" }, secrets: { diff --git a/packages/server/src/__tests__/build-app-validation.test.ts b/packages/server/src/__tests__/build-app-validation.test.ts index 508624119..7dfbe6ed5 100644 --- a/packages/server/src/__tests__/build-app-validation.test.ts +++ b/packages/server/src/__tests__/build-app-validation.test.ts @@ -26,6 +26,15 @@ const baseConfig: Config = { docs: { enabled: false }, cronJobs: { enabled: false }, database: { url: process.env.DATABASE_URL ?? "", provider: "external" }, + attachments: { + downloadMode: "proxy", + orgQuotaBytes: 2 * 1024 * 1024 * 1024, + orgQuotaCount: 1000, + sweepIntervalSeconds: 0, + orphanGraceSeconds: 86_400, + pendingTtlSeconds: 3600, + maxConcurrentUploadsPerUploader: 4, + }, server: { port: 0, host: "127.0.0.1", publicUrl: undefined }, workspace: { root: "/tmp/first-tree-test-workspaces" }, secrets: { diff --git a/packages/server/src/__tests__/chat-message-post-shape.test.ts b/packages/server/src/__tests__/chat-message-post-shape.test.ts index 70b565653..e97ccb2a1 100644 --- a/packages/server/src/__tests__/chat-message-post-shape.test.ts +++ b/packages/server/src/__tests__/chat-message-post-shape.test.ts @@ -4,7 +4,7 @@ import { describe, expect, it } from "vitest"; import { createAgent } from "../services/agent.js"; import { createMeChat } from "../services/me-chat.js"; import { sendMessage } from "../services/message.js"; -import { createTestAdmin, useTestApp } from "./helpers.js"; +import { createTestAdmin, useTestApp, workerObjectStorage } from "./helpers.js"; /** * Wire-level shape of `POST /api/v1/chats/:chatId/messages` (the web @@ -18,7 +18,7 @@ import { createTestAdmin, useTestApp } from "./helpers.js"; * request flips back to open and a threaded reply unthreads (PR 981 review). */ describe("POST /chats/:chatId/messages — response carries metadata + inReplyTo", () => { - const getApp = useTestApp(); + const getApp = useTestApp({ objectStorage: workerObjectStorage() }); async function uploadAttachment( app: FastifyInstance, diff --git a/packages/server/src/__tests__/global-setup.ts b/packages/server/src/__tests__/global-setup.ts index 3770d4882..28c10c110 100644 --- a/packages/server/src/__tests__/global-setup.ts +++ b/packages/server/src/__tests__/global-setup.ts @@ -1,9 +1,46 @@ import { execSync } from "node:child_process"; +import { CreateBucketCommand, S3Client } from "@aws-sdk/client-s3"; import { PostgreSqlContainer } from "@testcontainers/postgresql"; import postgres from "postgres"; -import { MAX_FORKS, TEMPLATE_DB, WORKER_DB_PREFIX } from "./test-config.js"; +import { GenericContainer, Wait } from "testcontainers"; +import { MAX_FORKS, TEMPLATE_DB, WORKER_DB_PREFIX, WORKER_S3_BUCKET_PREFIX } from "./test-config.js"; let container: Awaited> | undefined; +let minioContainer: Awaited> | undefined; + +/** + * Pinned MinIO release so local and CI runs exercise the same S3 dialect. + * Suites never require real cloud credentials — this container (or the + * `CI_S3_ENDPOINT` escape hatch) is the only storage backend tests touch. + */ +const MINIO_IMAGE = "minio/minio:RELEASE.2025-04-22T22-12-26Z"; +const MINIO_ROOT_USER = "vitest-minio"; +const MINIO_ROOT_PASSWORD = "vitest-minio-secret"; + +async function startObjectStorage(): Promise<{ endpoint: string; accessKeyId: string; secretAccessKey: string }> { + // Escape hatch mirroring CI_DATABASE_URL: point tests at an externally + // provisioned S3-compatible server (e.g. a CI sidecar) instead of the + // testcontainers-managed MinIO. + const ciEndpoint = process.env.CI_S3_ENDPOINT; + if (ciEndpoint) { + return { + endpoint: ciEndpoint, + accessKeyId: process.env.CI_S3_ACCESS_KEY_ID ?? MINIO_ROOT_USER, + secretAccessKey: process.env.CI_S3_SECRET_ACCESS_KEY ?? MINIO_ROOT_PASSWORD, + }; + } + minioContainer = await new GenericContainer(MINIO_IMAGE) + .withEnvironment({ MINIO_ROOT_USER, MINIO_ROOT_PASSWORD }) + .withCommand(["server", "/data"]) + .withExposedPorts(9000) + .withWaitStrategy(Wait.forHttp("/minio/health/ready", 9000)) + .start(); + return { + endpoint: `http://${minioContainer.getHost()}:${minioContainer.getMappedPort(9000)}`, + accessKeyId: MINIO_ROOT_USER, + secretAccessKey: MINIO_ROOT_PASSWORD, + }; +} export async function setup() { // CI fast path: a sidecar Postgres is already running (GitHub Actions @@ -13,6 +50,8 @@ export async function setup() { // back to testcontainers as before. const ciUrl = process.env.CI_DATABASE_URL; let baseUrl: string; + // MinIO starts concurrently with PG — neither depends on the other. + const objectStoragePromise = startObjectStorage(); if (ciUrl) { baseUrl = ciUrl; } else { @@ -52,10 +91,40 @@ export async function setup() { await admin.end(); } + // Per-worker buckets mirror the per-worker databases: file-parallel + // workers write to disjoint buckets, and object keys are UUID-derived so + // suites never collide within a worker either. + const objectStorage = await objectStoragePromise; + const s3 = new S3Client({ + region: "us-east-1", + endpoint: objectStorage.endpoint, + forcePathStyle: true, + credentials: { + accessKeyId: objectStorage.accessKeyId, + secretAccessKey: objectStorage.secretAccessKey, + }, + }); + try { + for (let i = 1; i <= MAX_FORKS; i++) { + try { + await s3.send(new CreateBucketCommand({ Bucket: `${WORKER_S3_BUCKET_PREFIX}${i}` })); + } catch (error) { + // Re-runs against a live CI sidecar hit BucketAlreadyOwnedByYou. + const name = typeof error === "object" && error !== null && "name" in error ? error.name : undefined; + if (name !== "BucketAlreadyOwnedByYou" && name !== "BucketAlreadyExists") throw error; + } + } + } finally { + s3.destroy(); + } + // Hand-off to per-worker setup.ts via env (workers inherit parent env at // spawn under the `forks` pool). process.env.VITEST_PG_BASE_URL = baseUrl; process.env.VITEST_PG_MAX_WORKERS = String(MAX_FORKS); + process.env.VITEST_S3_ENDPOINT = objectStorage.endpoint; + process.env.VITEST_S3_ACCESS_KEY_ID = objectStorage.accessKeyId; + process.env.VITEST_S3_SECRET_ACCESS_KEY = objectStorage.secretAccessKey; // Leave DATABASE_URL pointing at the template until setup.ts replaces it // per-worker; nothing reads DATABASE_URL between globalSetup and worker // bootstrap, so this is just a sane default if that ever changes. @@ -66,4 +135,5 @@ export async function setup() { export async function teardown() { await container?.stop(); + await minioContainer?.stop(); } diff --git a/packages/server/src/__tests__/helpers.ts b/packages/server/src/__tests__/helpers.ts index 5d37dc754..6897aa272 100644 --- a/packages/server/src/__tests__/helpers.ts +++ b/packages/server/src/__tests__/helpers.ts @@ -76,6 +76,12 @@ export type CreateTestAppOptions = { rateLimit?: Partial>; connectBootstrap?: Config["connectBootstrap"]; inbox?: Partial>; + /** + * Object storage config. Absent by default — suites that exercise the S3 + * surface pass the per-worker MinIO target provisioned by global setup. + */ + objectStorage?: Config["objectStorage"]; + attachments?: Partial; runtimeHttpTokenEnforcement?: boolean; runtimeSwitchFaultInjection?: boolean; allowedOrganizationId?: string; @@ -98,6 +104,31 @@ export type CreateTestAppOptions = { githubAppPrivateKeyPem?: string; }; +/** + * Object-storage config pointing at this worker's MinIO bucket (provisioned + * by global-setup, one bucket per pool slot — the S3 analogue of the + * per-worker database). Suites that exercise the attachment S3 surface pass + * the result as `createTestApp({ objectStorage: workerObjectStorage() })`. + */ +export function workerObjectStorage(): NonNullable { + const endpoint = process.env.VITEST_S3_ENDPOINT; + if (!endpoint) { + throw new Error("VITEST_S3_ENDPOINT not set — vitest global setup did not provision object storage"); + } + const maxWorkers = Number.parseInt(process.env.VITEST_PG_MAX_WORKERS ?? "1", 10); + const rawId = Number.parseInt(process.env.VITEST_POOL_ID ?? "1", 10); + const slot = ((rawId - 1) % Math.max(1, maxWorkers)) + 1; + return { + bucket: `vitest-w${slot}`, + accessKeyId: process.env.VITEST_S3_ACCESS_KEY_ID ?? "", + secretAccessKey: process.env.VITEST_S3_SECRET_ACCESS_KEY ?? "", + endpoint, + region: "us-east-1", + forcePathStyle: true, + publicEndpoint: undefined, + }; +} + export async function createTestApp(opts: CreateTestAppOptions = {}): Promise { const baseRateLimit = { max: 10000, @@ -136,6 +167,19 @@ export async function createTestApp(opts: CreateTestAppOptions = {}): Promise { await ackAll(app, b2Before, b2.inboxId); const { editMessage } = await import("../services/message.js"); - await editMessage(app.db, c1.id, m1.message.id, b1.uuid, { content: "first cut — revised" }); + await editMessage(app.db, null, c1.id, m1.message.id, b1.uuid, { content: "first cut — revised" }); // No new inbox entries should have been written — b2 pulls nothing. const b2After = await pollInbox(app.db, b2.inboxId, 10); diff --git a/packages/server/src/__tests__/open-question.test.ts b/packages/server/src/__tests__/open-question.test.ts index bb0829c82..3ad6b43a3 100644 --- a/packages/server/src/__tests__/open-question.test.ts +++ b/packages/server/src/__tests__/open-question.test.ts @@ -754,14 +754,16 @@ describe("open-question (format=request) + open_request_count", () => { metadata: { mentions: [other.uuid] }, }); - await expect(editMessage(app.db, chat.id, question.id, asker.agent.uuid, { format: "text" })).rejects.toThrow( + await expect(editMessage(app.db, null, chat.id, question.id, asker.agent.uuid, { format: "text" })).rejects.toThrow( /format to or from 'request'/i, ); - await expect(editMessage(app.db, chat.id, plain.id, asker.agent.uuid, { format: "request" })).rejects.toThrow( + await expect(editMessage(app.db, null, chat.id, plain.id, asker.agent.uuid, { format: "request" })).rejects.toThrow( /format to or from 'request'/i, ); // A content-only edit of the request is still allowed. - const edited = await editMessage(app.db, chat.id, question.id, asker.agent.uuid, { content: "ratio (clarified)?" }); + const edited = await editMessage(app.db, null, chat.id, question.id, asker.agent.uuid, { + content: "ratio (clarified)?", + }); expect(edited.format).toBe("request"); }); @@ -780,11 +782,11 @@ describe("open-question (format=request) + open_request_count", () => { metadata: { mentions: [human.uuid], request: { question: "5% or 20%?" } }, }); - await expect(editMessage(app.db, chat.id, question.id, asker.agent.uuid, { content: " " })).rejects.toThrow( + await expect(editMessage(app.db, null, chat.id, question.id, asker.agent.uuid, { content: " " })).rejects.toThrow( BadRequestError, ); await expect( - editMessage(app.db, chat.id, question.id, asker.agent.uuid, { content: "PLACEHOLDER" }), + editMessage(app.db, null, chat.id, question.id, asker.agent.uuid, { content: "PLACEHOLDER" }), ).rejects.toThrow(BadRequestError); }); }); diff --git a/packages/server/src/__tests__/setup.ts b/packages/server/src/__tests__/setup.ts index ae2511230..f6e95b805 100644 --- a/packages/server/src/__tests__/setup.ts +++ b/packages/server/src/__tests__/setup.ts @@ -63,6 +63,8 @@ const TRUNCATE_TABLES = [ "inbox_entries", "session_events", "notifications", + "attachment_references", + "attachments", "messages", "chat_user_state", "chat_membership", diff --git a/packages/server/src/__tests__/stream-limit.test.ts b/packages/server/src/__tests__/stream-limit.test.ts new file mode 100644 index 000000000..1562ca3de --- /dev/null +++ b/packages/server/src/__tests__/stream-limit.test.ts @@ -0,0 +1,148 @@ +import { PassThrough, Readable } from "node:stream"; +import { pipeline } from "node:stream/promises"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { createByteLimitStream, settleStreamingUpload } from "../services/stream-limit.js"; + +function limiterFor(expectedBytes: number) { + return createByteLimitStream({ + expectedBytes, + makeMismatchError: (seen) => new Error(`mismatch: declared ${expectedBytes}, saw ${seen}`), + }); +} + +async function drain(stream: Readable): Promise { + const chunks: Buffer[] = []; + for await (const chunk of stream) { + chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)); + } + return Buffer.concat(chunks); +} + +describe("createByteLimitStream", () => { + it("passes an exact-length body through unchanged", async () => { + const limiter = limiterFor(6); + const [collected] = await Promise.all([drain(limiter), pipeline(Readable.from([Buffer.from("sixby!")]), limiter)]); + expect(collected.toString()).toBe("sixby!"); + }); + + it("fails mid-stream when the body exceeds the declared length", async () => { + const limiter = limiterFor(3); + await expect( + Promise.all([ + drain(limiter).catch(() => Buffer.alloc(0)), + pipeline(Readable.from([Buffer.from("sixby!")]), limiter), + ]), + ).rejects.toThrow(/mismatch: declared 3, saw 6/); + }); + + it("fails at EOF when the body undershoots the declared length", async () => { + const limiter = limiterFor(10); + await expect( + Promise.all([ + drain(limiter).catch(() => Buffer.alloc(0)), + pipeline(Readable.from([Buffer.from("shrt")]), limiter), + ]), + ).rejects.toThrow(/mismatch: declared 10, saw 4/); + }); +}); + +describe("settleStreamingUpload — cross-cancel coordination", () => { + // The whole point of the coordinator (and of commit "cross-cancel + // streaming upload halves") is that NO failure mode may orphan a + // rejection. Collect any unhandled rejections that surface between test + // start and a post-test settle window, and assert zero. + const orphans: unknown[] = []; + const collect = (reason: unknown) => { + orphans.push(reason); + }; + + beforeEach(() => { + orphans.length = 0; + process.on("unhandledRejection", collect); + }); + + afterEach(async () => { + // Give any stray rejection two macrotask turns to surface. + await new Promise((resolve) => setImmediate(resolve)); + await new Promise((resolve) => setImmediate(resolve)); + process.off("unhandledRejection", collect); + expect(orphans).toEqual([]); + }); + + it("producer failure aborts the consumer and surfaces the producer error", async () => { + const limiter = limiterFor(3); + let observedSignal: AbortSignal | undefined; + const consumerSettled: string[] = []; + + await expect( + settleStreamingUpload({ + limiter, + producer: pipeline(Readable.from([Buffer.from("sixby!")]), limiter), + startConsumer: (abortSignal) => { + observedSignal = abortSignal; + return new Promise((_resolve, reject) => { + abortSignal.addEventListener("abort", () => { + consumerSettled.push("aborted"); + reject(new Error("consumer aborted")); + }); + }); + }, + }), + ).rejects.toThrow(/mismatch: declared 3, saw 6/); + + expect(observedSignal?.aborted).toBe(true); + expect(consumerSettled).toEqual(["aborted"]); + }); + + it("consumer failure destroys the limiter so the backpressured producer settles", async () => { + const limiter = limiterFor(1024); + // A source that never ends: without the cross-cancel, pipeline() would + // wait forever once the consumer stops reading. + const source = new PassThrough(); + source.write(Buffer.from("partial")); + + await expect( + settleStreamingUpload({ + limiter, + producer: pipeline(source, limiter), + startConsumer: async () => { + throw new Error("storage down"); + }, + }), + ).rejects.toThrow(/storage down/); + + expect(limiter.destroyed).toBe(true); + source.destroy(); + }); + + it("prefers the producer error when both halves reject", async () => { + const limiter = limiterFor(3); + await expect( + settleStreamingUpload({ + limiter, + producer: pipeline(Readable.from([Buffer.from("sixby!")]), limiter), + // The consumer fails with its OWN error once the abort reaches it — + // a genuine double rejection; the mismatch (root cause) must win. + startConsumer: (abortSignal) => + new Promise((_resolve, reject) => { + abortSignal.addEventListener("abort", () => reject(new Error("storage down"))); + }), + }), + ).rejects.toThrow(/mismatch: declared 3/); + }); + + it("resolves cleanly when both halves succeed", async () => { + const limiter = limiterFor(6); + await expect( + settleStreamingUpload({ + limiter, + producer: pipeline(Readable.from([Buffer.from("sixby!")]), limiter), + startConsumer: async (abortSignal) => { + const collected = await drain(limiter); + expect(collected.toString()).toBe("sixby!"); + expect(abortSignal.aborted).toBe(false); + }, + }), + ).resolves.toBeUndefined(); + }); +}); diff --git a/packages/server/src/__tests__/test-config.ts b/packages/server/src/__tests__/test-config.ts index 525f2dc1a..1e7ef17dc 100644 --- a/packages/server/src/__tests__/test-config.ts +++ b/packages/server/src/__tests__/test-config.ts @@ -15,3 +15,5 @@ export const MAX_FORKS = Number.isFinite(envCap) && envCap > 0 ? envCap : isCi ? export const WORKER_DB_PREFIX = "vitest_w"; export const TEMPLATE_DB = "vitest_template"; +// S3 bucket names must be DNS-safe: lowercase + hyphens (no underscores). +export const WORKER_S3_BUCKET_PREFIX = "vitest-w"; diff --git a/packages/server/src/api/agent/messages.ts b/packages/server/src/api/agent/messages.ts index 780999920..9dd724614 100644 --- a/packages/server/src/api/agent/messages.ts +++ b/packages/server/src/api/agent/messages.ts @@ -84,6 +84,7 @@ export async function agentMessageRoutes(app: FastifyInstance): Promise { const body = editMessageSchema.parse(request.body); const msg = await messageService.editMessage( app.db, + app.objectStorage, request.params.chatId, request.params.messageId, identity.uuid, diff --git a/packages/server/src/api/agents.ts b/packages/server/src/api/agents.ts index a665bffcc..d419fce6e 100644 --- a/packages/server/src/api/agents.ts +++ b/packages/server/src/api/agents.ts @@ -1,4 +1,6 @@ +import { Readable } from "node:stream"; import { + ATTACHMENT_ERROR_CODES, agentPinnedMessageSchema, switchAgentRuntimeSchema, updateAgentSchema, @@ -6,7 +8,7 @@ import { } from "@first-tree/shared"; import { getServerCliBinding } from "@first-tree/shared/channel"; import type { FastifyInstance, FastifyRequest } from "fastify"; -import { BadRequestError, ForbiddenError } from "../errors.js"; +import { BadRequestError, ForbiddenError, LengthRequiredError } from "../errors.js"; import { assertAllAgentsVisibleInOrg, requireAgentAccess } from "../scope/require-resource.js"; import * as agentService from "../services/agent.js"; import { @@ -286,46 +288,55 @@ export async function agentRoutes(app: FastifyInstance): Promise { // ─── Avatar image (M2) ────────────────────────────────────────────── // - // PUT accepts the raw image bytes as `application/octet-stream` / - // `image/*` so we don't have to pull in `@fastify/multipart` for a - // single-field upload. The web client always pre-resizes to ~50KB WEBP; - // server enforces ≤ MAX_AVATAR_IMAGE_BYTES regardless. + // PUT accepts the raw image bytes as `image/*` so we don't have to pull + // in `@fastify/multipart` for a single-field upload. The parser hands + // the raw stream through and the service pipes it straight to object + // storage — the payload is never buffered in Node. Content-Length is + // required (411): the ≤ MAX_AVATAR_IMAGE_BYTES cap is enforced on the + // declared size before the body is consumed, then byte-exactly during + // the stream. The web client always pre-resizes to ~50KB WEBP. // // GET is intentionally public: `` cannot send the Authorization // header. The agent UUID is unguessable v7, and the surrounding ACL on // /api/v1/agents already keeps the UUID itself off public surfaces. - app.addContentTypeParser(/^image\//, { parseAs: "buffer" }, (_req, body, done) => { - done(null, body); + app.addContentTypeParser(/^image\//, (_req, payload, done) => { + done(null, payload); }); - app.put<{ Params: { uuid: string } }>( - "/:uuid/avatar", - { bodyLimit: agentService.MAX_AVATAR_IMAGE_BYTES + 1024 }, - async (request, reply) => { - const { agent } = await requireAgentAccess(request, app.db, "manage"); - assertMutableAgentIsNotLandingCampaignTrial(agent); - const contentType = request.headers["content-type"]; - if (typeof contentType !== "string" || !contentType.startsWith("image/")) { - throw new BadRequestError( - `Avatar upload requires an image/* Content-Type. Supported: ${SUPPORTED_AVATAR_IMAGE_MIMES.join(", ")}.`, - ); - } - const mime = contentType.split(";")[0]?.trim() ?? ""; - const body = request.body; - if (!Buffer.isBuffer(body)) { - throw new BadRequestError("Avatar upload body must be raw image bytes."); - } - const updatedAt = await agentService.setAgentAvatarImage(app.db, request.params.uuid, body, mime); - return reply.status(200).send({ - avatarImageUrl: agentAvatarImageUrl(request.params.uuid, updatedAt), + app.put<{ Params: { uuid: string } }>("/:uuid/avatar", async (request, reply) => { + const { agent } = await requireAgentAccess(request, app.db, "manage"); + assertMutableAgentIsNotLandingCampaignTrial(agent); + const contentType = request.headers["content-type"]; + if (typeof contentType !== "string" || !contentType.startsWith("image/")) { + throw new BadRequestError( + `Avatar upload requires an image/* Content-Type. Supported: ${SUPPORTED_AVATAR_IMAGE_MIMES.join(", ")}.`, + ); + } + const mime = contentType.split(";")[0]?.trim() ?? ""; + const rawLength = request.headers["content-length"]; + const contentLength = typeof rawLength === "string" ? Number.parseInt(rawLength, 10) : Number.NaN; + if (!Number.isFinite(contentLength) || contentLength < 0) { + throw new LengthRequiredError("Avatar uploads must declare Content-Length", { + code: ATTACHMENT_ERROR_CODES.lengthRequired, }); - }, - ); + } + const body = request.body; + if (!(body instanceof Readable)) { + throw new BadRequestError("Avatar upload body must be raw image bytes."); + } + const updatedAt = await agentService.setAgentAvatarImage(app.db, app.objectStorage, request.params.uuid, body, { + mime, + contentLength, + }); + return reply.status(200).send({ + avatarImageUrl: agentAvatarImageUrl(request.params.uuid, updatedAt), + }); + }); app.delete<{ Params: { uuid: string } }>("/:uuid/avatar", async (request, reply) => { const { agent } = await requireAgentAccess(request, app.db, "manage"); assertMutableAgentIsNotLandingCampaignTrial(agent); - await agentService.clearAgentAvatarImage(app.db, request.params.uuid); + await agentService.clearAgentAvatarImage(app.db, app.objectStorage, request.params.uuid); return reply.status(204).send(); }); @@ -481,9 +492,46 @@ export async function publicAgentAvatarRoutes(app: FastifyInstance): Promise` suffix on the URL, // so immutable + 30d is safe and avoids round-trips from chat surfaces // that render the image hundreds of times per session. - reply.header("Content-Type", image.mime); reply.header("Cache-Control", "public, max-age=2592000, immutable"); reply.header("ETag", `"${image.updatedAt.getTime()}"`); - return reply.send(image.data); + + // Transitional branch: payload still inline in PG (pre-migration row). + if (image.data) { + reply.header("Content-Type", image.mime); + return reply.send(image.data); + } + + const objectStorage = app.objectStorage; + if (!objectStorage || !image.objectKey) { + // objectKey is set whenever data is absent; a missing storage config + // for a migrated avatar is a deployment gap, not a client error. + request.log.error( + { uuid: request.params.uuid }, + "avatar payload is in object storage but storage is unavailable", + ); + return reply.status(503).send({ error: "Avatar storage unavailable" }); + } + + // Avatars are always proxied, regardless of `attachments.downloadMode`: + // a 302 to a presigned URL varies per request (fresh signature), which + // would defeat the immutable browser cache this surface depends on — + // chat surfaces render the same avatar hundreds of times per session. + // The payload is ≤ 512 KiB and streamed, so proxying stays cheap. + const object = await objectStorage.getObjectStream(image.objectKey); + if (!object) { + request.log.error( + { uuid: request.params.uuid, objectKey: image.objectKey }, + "avatar payload missing from object storage", + ); + return reply.status(404).send({ error: "Avatar not set" }); + } + reply.header("Content-Type", image.mime); + if (object.contentLength !== undefined) { + reply.header("Content-Length", object.contentLength); + } + reply.raw.once("close", () => { + object.body.destroy(); + }); + return reply.send(object.body); }); } diff --git a/packages/server/src/api/attachments.ts b/packages/server/src/api/attachments.ts index fe465c046..5e93fd395 100644 --- a/packages/server/src/api/attachments.ts +++ b/packages/server/src/api/attachments.ts @@ -1,6 +1,7 @@ -import type { FastifyInstance } from "fastify"; -import { NotFoundError } from "../errors.js"; +import type { FastifyInstance, FastifyReply } from "fastify"; +import { NotFoundError, ServiceUnavailableError } from "../errors.js"; import { loadAttachmentData, loadAttachmentMeta } from "../services/attachment.js"; +import { attachmentObjectKey, contentDisposition } from "../services/object-storage.js"; /** * Object-storage primitive — download surface. @@ -16,18 +17,31 @@ import { loadAttachmentData, loadAttachmentMeta } from "../services/attachment.j * authorization layers it on top (e.g. an image message hands the id only to * its legitimate recipients); the primitive deliberately stays thin. * + * Serving modes (config `attachments.downloadMode`): + * + * - `proxy` (default): the payload is piped from object storage through the + * server — no full-object buffering, works with any bucket topology. + * - `redirect`: 302 to a short-lived presigned URL. Cheaper at scale but + * requires a browser-reachable bucket with CORS for the web origin. + * + * Rows the migration command has not moved yet still carry the payload in + * the legacy `bytea` column and are served from PG directly (bounded by the + * 10 MiB cap); a row migrated between the metadata read and the payload + * read falls through to object storage via the deterministic key. + * * Upload lives separately at `POST /api/v1/orgs/:orgId/attachments` * (Class B) because the uploader identity is org-scoped — see * api/orgs/attachments.ts. */ export async function attachmentRoutes(app: FastifyInstance): Promise { - app.get<{ Params: { id: string } }>("/:id", async (request, reply) => { + app.get<{ Params: { id: string } }>("/:id", { config: { otelRecordBody: false } }, async (request, reply) => { const { id } = request.params; // Load metadata only — the ETag check runs off this, so a 304 cache hit - // never drags the bytea payload out of PG. + // never touches the payload (in PG or object storage). const meta = await loadAttachmentMeta(app.db, id); - if (!meta) { + if (!meta || meta.state !== "stored") { + // `pending` uploads and `deleting` tombstones are not observable. throw new NotFoundError(`Attachment "${id}" not found`); } @@ -39,30 +53,59 @@ export async function attachmentRoutes(app: FastifyInstance): Promise { return reply.status(304).send(); } - const data = await loadAttachmentData(app.db, id); - if (!data) { - // Deleted between the metadata read and now — vanishingly rare, but - // surface it honestly rather than streaming an empty body. - throw new NotFoundError(`Attachment "${id}" not found`); + // Transitional branch: payload still inline in PG (pre-migration row). + if (!meta.objectKey) { + const data = await loadAttachmentData(app.db, id); + if (data) { + setPayloadHeaders(reply, meta.mimeType, meta.filename, etag); + reply.header("Content-Length", meta.sizeBytes); + return reply.send(data); + } + // Migrated between the two reads — fall through to object storage on + // the deterministic key. + } + + const objectStorage = app.objectStorage; + if (!objectStorage) { + throw new ServiceUnavailableError( + "Object storage is not configured (FIRST_TREE_S3_*); this attachment's payload has been migrated and cannot be served", + ); } + const objectKey = meta.objectKey ?? attachmentObjectKey(meta.id); - reply - .header("Content-Type", meta.mimeType) - .header("Content-Length", meta.sizeBytes) - .header("Cache-Control", "private, max-age=31536000, immutable") - .header("ETag", etag) - .header("X-Content-Type-Options", "nosniff") - .header("Content-Disposition", `inline; filename="${encodeRfc6266Filename(meta.filename)}"`); - return reply.send(data); + if (app.config.attachments.downloadMode === "redirect") { + const url = await objectStorage.presignGetUrl(objectKey, { + filename: meta.filename, + mimeType: meta.mimeType, + disposition: "inline", + }); + // The presigned URL itself is the short-lived secret — never cache it. + return reply.status(302).header("Cache-Control", "private, no-store").header("Location", url).send(); + } + + const object = await objectStorage.getObjectStream(objectKey); + if (!object) { + // A `stored` row without its object is corruption — be loud in logs, + // honest (404) on the wire. + request.log.error({ attachmentId: id, objectKey }, "stored attachment payload missing from object storage"); + throw new NotFoundError(`Attachment "${id}" not found`); + } + setPayloadHeaders(reply, meta.mimeType, meta.filename, etag); + reply.header("Content-Length", meta.sizeBytes); + // Fastify destroys a streamed body when the client goes away, but only + // once it starts reading; cover the pre-read abort window too. + reply.raw.once("close", () => { + object.body.destroy(); + }); + return reply.send(object.body); }); } -/** - * RFC 6266 percent-encoding for the `filename` directive — `inline; filename="..."`. - * Only percent-encodes characters that would break the quoted-string parser - * (CR/LF, quote, backslash). Browsers tolerate non-ASCII inside the quoted - * form, but raw quotes / control chars would smuggle headers. - */ -function encodeRfc6266Filename(name: string): string { - return name.replace(/[\r\n"\\]/g, (c) => `%${c.charCodeAt(0).toString(16).toUpperCase().padStart(2, "0")}`); +function setPayloadHeaders(reply: FastifyReply, mimeType: string, filename: string, etag: string): void { + reply + .header("Content-Type", mimeType) + .header("Cache-Control", "private, max-age=31536000, immutable") + .header("ETag", etag) + .header("X-Content-Type-Options", "nosniff") + .header("Content-Disposition", contentDisposition(filename, "inline")); } diff --git a/packages/server/src/api/orgs/attachments.ts b/packages/server/src/api/orgs/attachments.ts index ae0570206..18ff57bbb 100644 --- a/packages/server/src/api/orgs/attachments.ts +++ b/packages/server/src/api/orgs/attachments.ts @@ -1,44 +1,74 @@ +import { Readable } from "node:stream"; +import { pipeline } from "node:stream/promises"; import { + ATTACHMENT_ERROR_CODES, ATTACHMENT_FILENAME_HEADER, ATTACHMENT_MIME_HEADER, MAX_ATTACHMENT_BYTES, type UploadAttachmentResponse, } from "@first-tree/shared"; import type { FastifyInstance } from "fastify"; -import { BadRequestError } from "../../errors.js"; +import { + BadRequestError, + ConflictError, + LengthRequiredError, + PayloadTooLargeError, + ServiceUnavailableError, +} from "../../errors.js"; import { requireOrgMembership } from "../../scope/require-org.js"; -import { createAttachment } from "../../services/attachment.js"; +import { deletePendingReservation, finalizeAttachment, reserveAttachment } from "../../services/attachment.js"; +import { createByteLimitStream, settleStreamingUpload } from "../../services/stream-limit.js"; +import { createUploadGate } from "../../services/upload-gate.js"; /** * Object-storage primitive — upload surface (Class B, org-scoped). * * POST /api/v1/orgs/:orgId/attachments — upload bytes * - * Org-scoped because `uploaded_by` must resolve to a stable team identity: - * `requireOrgMembership` reads the caller's active member in `:orgId` and - * yields their `humanAgentId` deterministically. Putting the org in the path - * (rather than a `?orgId=` query) follows the repo's HTTP path convention and - * removes the multi-org ambiguity a query param would leave open. + * Org-scoped because `uploaded_by` must resolve to a stable team identity + * and quota accounting needs an owning org: `requireOrgMembership` reads + * the caller's active member in `:orgId` and yields their `humanAgentId` + * deterministically. * * Upload protocol: `Content-Type: application/octet-stream` body + the * `x-attachment-mime` / `x-attachment-filename` headers carry the logical - * metadata. This keeps the body parser uniform and avoids a multipart - * dependency. The byte cap is enforced both as a route `bodyLimit` and again - * in the service layer. + * metadata. `Content-Length` is REQUIRED (411 otherwise): the org quota is + * reserved from the declared size before the body is consumed, and the + * byte-limit stream enforces the declaration exactly. + * + * The body is never materialized in memory: a scoped content-type parser + * hands the raw request stream through, and it is piped straight into the + * object-storage PUT. Flow per request: + * + * 1. membership + header validation (before touching the body) + * 2. per-uploader concurrency gate → 429 ATTACHMENT_CONCURRENCY_EXCEEDED + * 3. quota reservation (pending row) → 413 / 422 ATTACHMENT_QUOTA_EXCEEDED + * 4. stream to object storage (byte-limited) + * 5. finalize: pending → stored CAS; a reservation reclaimed by the + * pending-TTL sweep mid-upload surfaces as 409 + * + * Any failure after (3) best-effort deletes both the object and the + * pending row; a crash instead leaves the reservation to the sweep. * * Download lives separately at `GET /api/v1/attachments/:id` — see * api/attachments.ts. */ export async function orgAttachmentRoutes(app: FastifyInstance): Promise { - // Single value for both client visibility (415 if wrong type) and route - // bodyLimit. 16 KB headroom beyond MAX_ATTACHMENT_BYTES leaves space for - // misc small request overhead without inflating the cap. - const UPLOAD_BODY_LIMIT = MAX_ATTACHMENT_BYTES + 16 * 1024; + // Plugin-scoped override of the global buffering octet-stream parser + // (app.ts): this surface wants the raw request stream — buffering would + // defeat the whole streaming path. Fastify clones parent parsers into the + // encapsulated context, so the inherited one must be removed before the + // passthrough can register; the override ends at this plugin's boundary. + app.removeContentTypeParser("application/octet-stream"); + app.addContentTypeParser("application/octet-stream", (_request, payload, done) => { + done(null, payload); + }); + + const uploadGate = createUploadGate(app.config.attachments.maxConcurrentUploadsPerUploader); app.post<{ Params: { orgId: string } }>( "/", { - bodyLimit: UPLOAD_BODY_LIMIT, config: { otelRecordBody: false }, }, async (request, reply) => { @@ -54,9 +84,29 @@ export async function orgAttachmentRoutes(app: FastifyInstance): Promise { throw new BadRequestError(`Content-Type must be application/octet-stream (got "${contentType || "missing"}")`); } - const body = request.body; - if (!Buffer.isBuffer(body)) { - throw new BadRequestError("Request body must be raw bytes"); + const objectStorage = app.objectStorage; + if (!objectStorage) { + throw new ServiceUnavailableError( + "Object storage is not configured (FIRST_TREE_S3_*); attachment uploads are unavailable", + ); + } + + const rawLength = request.headers["content-length"]; + const contentLength = typeof rawLength === "string" ? Number.parseInt(rawLength, 10) : Number.NaN; + if (!Number.isFinite(contentLength) || contentLength < 0) { + // Quota is reserved from the declared size, so chunked transfer + // encoding cannot be admitted. + throw new LengthRequiredError("Attachment uploads must declare Content-Length", { + code: ATTACHMENT_ERROR_CODES.lengthRequired, + }); + } + if (contentLength === 0) { + throw new BadRequestError("Attachment is empty"); + } + if (contentLength > MAX_ATTACHMENT_BYTES) { + throw new PayloadTooLargeError(`Attachment exceeds maximum size of ${MAX_ATTACHMENT_BYTES} bytes`, { + code: ATTACHMENT_ERROR_CODES.tooLarge, + }); } const mimeHeader = request.headers[ATTACHMENT_MIME_HEADER]; @@ -68,22 +118,79 @@ export async function orgAttachmentRoutes(app: FastifyInstance): Promise { const filenameHeader = request.headers[ATTACHMENT_FILENAME_HEADER]; const filename = (Array.isArray(filenameHeader) ? filenameHeader[0] : filenameHeader)?.trim() || "blob"; - const row = await createAttachment(app.db, { - mimeType, - filename, - data: body, - uploadedBy: scope.humanAgentId, - }); + const body = request.body; + if (!(body instanceof Readable)) { + throw new BadRequestError("Request body must be raw bytes"); + } - const response: UploadAttachmentResponse = { - id: row.id, - mimeType: row.mimeType, - filename: row.filename, - sizeBytes: row.sizeBytes, - uploadedBy: row.uploadedBy, - createdAt: row.createdAt.toISOString(), - }; - return reply.status(201).send(response); + const releaseSlot = uploadGate.acquire(scope.humanAgentId); + try { + const reserved = await reserveAttachment(app.db, { + organizationId: scope.organizationId, + mimeType, + filename, + sizeBytes: contentLength, + uploadedBy: scope.humanAgentId, + quota: { + maxTotalBytes: app.config.attachments.orgQuotaBytes, + maxObjectCount: app.config.attachments.orgQuotaCount, + }, + }); + const objectKey = reserved.objectKey; + if (!objectKey) { + throw new Error("Attachment reservation is missing its object key"); + } + + try { + const limiter = createByteLimitStream({ + expectedBytes: contentLength, + makeMismatchError: (seenBytes) => + new BadRequestError( + `Request body does not match Content-Length (declared ${contentLength}, saw ${seenBytes}${seenBytes > contentLength ? "+" : ""} bytes)`, + ), + }); + // The SDK consumes the limiter as the PUT body while pipeline() + // pumps the request stream through it; settleStreamingUpload + // cross-cancels the halves on failure and never orphans either + // rejection. + await settleStreamingUpload({ + limiter, + producer: pipeline(body, limiter), + startConsumer: (abortSignal) => + objectStorage.putObjectStream(objectKey, limiter, { + contentLength, + contentType: mimeType, + abortSignal, + }), + }); + } catch (error) { + // Best-effort rollback; a crash instead leaves the pending row to + // the TTL sweep, which deletes object + row idempotently. + await objectStorage.deleteObject(objectKey).catch(() => {}); + await deletePendingReservation(app.db, reserved.id).catch(() => {}); + throw error; + } + + const finalized = await finalizeAttachment(app.db, reserved.id); + if (!finalized) { + // The upload outlived the pending TTL and the sweep reclaimed the + // reservation. The object was just written — remove it again. + await objectStorage.deleteObject(objectKey).catch(() => {}); + throw new ConflictError("Attachment upload exceeded the reservation window; retry the upload"); + } + + const response: UploadAttachmentResponse = { + id: reserved.id, + mimeType: reserved.mimeType, + filename: reserved.filename, + sizeBytes: reserved.sizeBytes, + uploadedBy: reserved.uploadedBy, + createdAt: reserved.createdAt.toISOString(), + }; + return reply.status(201).send(response); + } finally { + releaseSlot(); + } }, ); } diff --git a/packages/server/src/app.ts b/packages/server/src/app.ts index 2161ac8a1..a090db6b2 100644 --- a/packages/server/src/app.ts +++ b/packages/server/src/app.ts @@ -100,6 +100,7 @@ import { createConfigService } from "./services/config-service.js"; import { backfillGitlabAttentionPairs } from "./services/gitlab-attention-backfill.js"; import { repairMembershipHumanMirrors } from "./services/membership.js"; import { createNotifier, type Notifier } from "./services/notifier.js"; +import { createObjectStorage } from "./services/object-storage.js"; import { ensureDefaultOrganization } from "./services/organization.js"; import { createPulseAggregator } from "./services/pulse-aggregator.js"; import { createResourcesService } from "./services/resources.js"; @@ -296,6 +297,14 @@ export async function buildApp(config: Config) { app.decorate("db", db); app.decorate("config", config); + // S3-compatible object storage for binary payloads. Optional at the + // config layer: when absent the attachment upload surface answers 503 + // while legacy bytea downloads keep working — deployments configure + // storage, then run `migrate:attachments` (see the objectStorage group in + // shared server-config). + const objectStorage = config.objectStorage ? createObjectStorage(config.objectStorage) : null; + app.decorate("objectStorage", objectStorage); + // Advisory Command-package version broadcast to every Client via the // `server:welcome` WS frame. The poller refreshes the advertised value // from the npm registry on `config.update.pollIntervalMinutes`, so the @@ -350,12 +359,12 @@ export async function buildApp(config: Config) { options: { maxPayload: config.ws?.maxPayload ?? 65_536 }, }); - // Body parser for `application/octet-stream` — needed by the attachment - // upload route. Fastify's built-in parsers cover json / text only; without - // this registration `request.body` would be undefined on an octet-stream - // POST and the route would 415. Registered globally because Fastify only - // supports global content-type parsers; the route still owns its own - // `bodyLimit` so the byte cap is route-local. + // Default body parser for `application/octet-stream`. Fastify's built-in + // parsers cover json / text only; without a registration an octet-stream + // POST would 415. The attachment upload plugin overrides this in its own + // encapsulated context with a stream passthrough (see + // api/orgs/attachments.ts) — this buffering default covers everything + // else. app.addContentTypeParser("application/octet-stream", { parseAs: "buffer" }, (_req, body, done) => { done(null, body); }); @@ -707,6 +716,15 @@ export async function buildApp(config: Config) { if (gitlabAttentionBackfill.paired > 0 || gitlabAttentionBackfill.legacyRouteOnly > 0) { app.log.info(gitlabAttentionBackfill, "gitlab attention pair backfill complete"); } + if (objectStorage) { + // Dev/test convenience + fail-fast diagnostics; never blocks boot + // (ensureBucket degrades to warnings internally). + await objectStorage.ensureBucket(); + } else { + app.log.warn( + "object storage is not configured (FIRST_TREE_S3_*); attachment uploads will be rejected with 503 until it is", + ); + } await notifier.start(); backgroundTasks.start(); pulseAggregator.start(); diff --git a/packages/server/src/db/schema/agents.ts b/packages/server/src/db/schema/agents.ts index 29a9c9bc2..16e9a4d14 100644 --- a/packages/server/src/db/schema/agents.ts +++ b/packages/server/src/db/schema/agents.ts @@ -80,6 +80,15 @@ export const agents = pgTable( * edit. NULL iff data is NULL. */ avatarImageUpdatedAt: timestamp("avatar_image_updated_at", { withTimezone: true }), + /** + * Object-storage key of the avatar image (`avatars/`, overwritten + * in place on re-upload). NULL when no avatar is set or the image still + * sits in the legacy `avatar_image_data` bytea (pre-migration). Avatars + * are NOT `attachments` rows: 1:1 with the agent, bounded by the 512 KiB + * cap, and replaced rather than accumulated — so they carry no quota or + * reference-lifecycle accounting. + */ + avatarObjectKey: text("avatar_object_key"), /** * Agent-reported skill (slash-command) list. Discovered by the daemon by * scanning the agent runtime's skill directories (~/.claude/skills, diff --git a/packages/server/src/db/schema/attachment-references.ts b/packages/server/src/db/schema/attachment-references.ts new file mode 100644 index 000000000..817479dc1 --- /dev/null +++ b/packages/server/src/db/schema/attachment-references.ts @@ -0,0 +1,44 @@ +import { index, pgTable, primaryKey, text, timestamp } from "drizzle-orm/pg-core"; +import { attachments } from "./attachments.js"; +import { messages } from "./messages.js"; + +/** + * Reference ledger: one row per (attachment, message) link, maintained in + * the same transaction as the message write. A message references an + * attachment when its `content` carries an `imageId` (single or batch + * form) or its `metadata.attachments[]` carries an `attachmentId` — + * `collectAttachmentIds` in services/attachment-references.ts is the single + * source of truth for that discovery. + * + * The ledger exists so lifecycle checks are O(1) lookups instead of + * whole-table jsonb scans: "delete on last reference removed" checks for + * remaining edges, and the orphan sweep treats zero-edge rows as candidates + * (with a full-scan verify veto as the safety net for rows uploaded before + * the backfill ran — see services/attachment-sweep.ts). + * + * FK discipline: `attachment_id` deliberately has NO cascade (default + * NO ACTION — functionally restrictive). Every legitimate row-delete path + * proves the edge count is zero before deleting the attachment, so a + * cascade could only ever fire on a logic bug deleting a still-referenced + * attachment — and would then silently destroy the evidence. NO ACTION + * turns that bug into a loud constraint violation. `message_id` has no + * cascade either: messages are immutable and never row-deleted (enforced + * today by inbound restrictive FKs). + */ +export const attachmentReferences = pgTable( + "attachment_references", + { + attachmentId: text("attachment_id") + .notNull() + .references(() => attachments.id), + messageId: text("message_id") + .notNull() + .references(() => messages.id), + createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), + }, + (table) => [ + primaryKey({ columns: [table.attachmentId, table.messageId] }), + /** Edit-time recompute: fetch the current edge set of one message. */ + index("attachment_references_message_id_idx").on(table.messageId), + ], +); diff --git a/packages/server/src/db/schema/attachments.ts b/packages/server/src/db/schema/attachments.ts index 9d4d6a9f1..a5bdf8c95 100644 --- a/packages/server/src/db/schema/attachments.ts +++ b/packages/server/src/db/schema/attachments.ts @@ -1,4 +1,5 @@ import { customType, index, integer, pgTable, text, timestamp } from "drizzle-orm/pg-core"; +import { organizations } from "./organizations.js"; /** * `bytea` column type. Drizzle ships pg primitives but not bytea out of the @@ -12,35 +13,78 @@ const bytea = customType<{ data: Buffer; driverData: Buffer }>({ }); /** - * Server-side blob storage primitive. The first-tree object-storage layer. + * Attachment lifecycle states. + * + * - `pending` — row inserted as a durable quota reservation; the payload is + * still streaming to object storage. Invisible to download, not + * referenceable, reclaimed by the sweep once older than the pending TTL. + * - `stored` — payload verified in object storage (or, transitionally, in + * the legacy `data` bytea). Downloadable and referenceable. + * - `deleting` — tombstone claimed for deletion (orphan sweep or + * last-reference-removed). Invisible and blocks new references; the + * object and row are removed next, and a crash in between is retried by + * the sweep (object deletion is idempotent). + */ +export const ATTACHMENT_STATES = ["pending", "stored", "deleting"] as const; +export type AttachmentState = (typeof ATTACHMENT_STATES)[number]; + +/** + * Attachment metadata. The binary payload lives in S3-compatible object + * storage under the deterministic key `attachments/`; PostgreSQL keeps + * metadata plus lifecycle state only. * * Independent blob — intentionally NO `chat_id` / `message_id` columns. * Upstream consumers (the `imageId` field inside `messages.content` jsonb, - * future bookmark metadata, agent avatar references) hold the - * `attachments.id` reference. One byte sequence, many consumers. + * `metadata.attachments[]` refs) hold the `attachments.id` reference; the + * `attachment_references` edge table records those links for O(1) + * lifecycle checks (see attachment-references.ts). * * Auth happens at the route layer as a capability model: download requires * a valid user JWT plus knowledge of the unguessable UUIDv4 id; there is no * per-attachment ACL. Stronger, attachment-scoped authorization is the * consumer's responsibility. Upload is org-scoped * (`POST /api/v1/orgs/:orgId/attachments`) so `uploaded_by` resolves to a - * stable member identity. + * stable member identity and quota accounting has an owner. * - * Lifecycle: write-once. v1 keeps every row forever. A refcount / - * orphan-sweep job is a follow-up only if storage growth demands it; it - * would have to scan every known upstream reference site. + * Lifecycle: three-state (see `ATTACHMENT_STATES`). Unreferenced `stored` + * rows older than the orphan grace window are deleted by the background + * sweep; removing the last remaining reference deletes immediately. */ export const attachments = pgTable( "attachments", { /** UUIDv4. Same value upstream references store. */ id: text("id").primaryKey(), + /** + * Owning organization — the quota accounting unit. Nullable only for + * legacy rows whose uploader row disappeared before the migration + * backfill ran; such rows are exempt from quota sums. New uploads + * always set it. + */ + organizationId: text("organization_id").references(() => organizations.id), /** MIME as declared by the uploader. v1 does not restrict. */ mimeType: text("mime_type").notNull(), filename: text("filename").notNull(), /** Server-measured byte length; clients do not get to lie about this. */ sizeBytes: integer("size_bytes").notNull(), - data: bytea("data").notNull(), + /** + * Object-storage key of the payload (`attachments/`). NULL only on + * legacy rows whose payload still sits in `data` — the migration + * command sets the key and clears the bytea in one statement. + */ + objectKey: text("object_key"), + /** + * Lifecycle state (`ATTACHMENT_STATES`). The column default exists ONLY + * so pre-existing rows became valid `stored` rows when the column was + * added — new code must always set `state` explicitly. + */ + state: text("state").$type().notNull().default("stored"), + /** + * Legacy inline payload. The `migrate:attachments` command moves it to + * object storage and NULLs it; dropping the column entirely is a + * follow-up once deployments have migrated. + */ + data: bytea("data"), /** * `agents.uuid` of the team member who uploaded these bytes. Humans * store their `humanAgentId`; AI agents store their own uuid. No FK — @@ -53,5 +97,9 @@ export const attachments = pgTable( (table) => [ index("attachments_uploaded_by_idx").on(table.uploadedBy), index("attachments_created_at_idx").on(table.createdAt), + /** Quota sums: `WHERE organization_id = $1 AND state IN ('pending','stored')`. */ + index("attachments_org_state_idx").on(table.organizationId, table.state), + /** Sweep scans: expired `pending`, orphan-aged `stored`, leftover `deleting`. */ + index("attachments_state_created_at_idx").on(table.state, table.createdAt), ], ); diff --git a/packages/server/src/db/schema/index.ts b/packages/server/src/db/schema/index.ts index f46f6ce76..32d595c4a 100644 --- a/packages/server/src/db/schema/index.ts +++ b/packages/server/src/db/schema/index.ts @@ -3,6 +3,7 @@ export { agentConfigs } from "./agent-configs.js"; export { agentPresence } from "./agent-presence.js"; export { agentResourceBindings } from "./agent-resource-bindings.js"; export { agents } from "./agents.js"; +export { attachmentReferences } from "./attachment-references.js"; export { attachments } from "./attachments.js"; // Legacy audit table — kept for drizzle-kit parity only (see file header); not used at runtime. export { attentions } from "./attentions.js"; diff --git a/packages/server/src/errors.ts b/packages/server/src/errors.ts index ac860df2c..22bc107ca 100644 --- a/packages/server/src/errors.ts +++ b/packages/server/src/errors.ts @@ -79,6 +79,42 @@ export class UnprocessableError extends AppError { } } +/** + * 413 — a single request body exceeds a server-enforced byte cap. Pass a + * stable `code` via attrs when the caller needs machine-readable identity + * (the central error handler serializes `attrs.code` into the body). + */ +export class PayloadTooLargeError extends AppError { + constructor(message = "Payload too large", attrs?: AppErrorAttrs) { + super(413, message, attrs); + this.name = "PayloadTooLargeError"; + } +} + +/** + * 411 — the request must declare `Content-Length` up front (streaming + * uploads reserve quota from the declared size before consuming the body, + * so chunked transfer encoding cannot be admitted). + */ +export class LengthRequiredError extends AppError { + constructor(message = "Content-Length required", attrs?: AppErrorAttrs) { + super(411, message, attrs); + this.name = "LengthRequiredError"; + } +} + +/** + * 429 — the caller holds too many concurrent in-flight operations. Distinct + * from @fastify/rate-limit's request-frequency 429: this guards sustained + * parallel streams, not request count per window. + */ +export class TooManyRequestsError extends AppError { + constructor(message = "Too many concurrent requests", attrs?: AppErrorAttrs) { + super(429, message, attrs); + this.name = "TooManyRequestsError"; + } +} + /** * 503 — an upstream dependency (GitHub API) is temporarily unreachable. * The operation was NOT performed; the caller may safely retry later. diff --git a/packages/server/src/services/agent.ts b/packages/server/src/services/agent.ts index 3b6840f5e..efb91da0a 100644 --- a/packages/server/src/services/agent.ts +++ b/packages/server/src/services/agent.ts @@ -1,3 +1,5 @@ +import type { Readable } from "node:stream"; +import { pipeline } from "node:stream/promises"; import type { AgentSkills, AgentType, @@ -29,7 +31,15 @@ import { clients } from "../db/schema/clients.js"; import { members } from "../db/schema/members.js"; import { organizations } from "../db/schema/organizations.js"; import { users } from "../db/schema/users.js"; -import { BadRequestError, ClientRetiredError, ConflictError, ForbiddenError, NotFoundError } from "../errors.js"; +import { + BadRequestError, + ClientRetiredError, + ConflictError, + ForbiddenError, + NotFoundError, + ServiceUnavailableError, +} from "../errors.js"; +import { createLogger } from "../observability/logger.js"; import type { OrgScope } from "../scope/types.js"; import { uuidv7 } from "../uuid.js"; import { @@ -37,7 +47,9 @@ import { agentNotLandingCampaignTrialCondition, agentVisibilityCondition, } from "./access-control.js"; +import { avatarObjectKey, type ObjectStorage } from "./object-storage.js"; import { resolveDefaultOrgId } from "./organization.js"; +import { createByteLimitStream, settleStreamingUpload } from "./stream-limit.js"; import { recomputeWatchersForAgent } from "./watcher.js"; /** @@ -1302,68 +1314,141 @@ export async function deleteAgent(db: Database, uuid: string) { export const SUPPORTED_AVATAR_IMAGE_MIMES = ["image/webp", "image/png", "image/jpeg"] as const; export type SupportedAvatarImageMime = (typeof SUPPORTED_AVATAR_IMAGE_MIMES)[number]; -/** Hard server-side ceiling for the stored bytea blob. Client pre-resizes to ~50KB. */ +/** Hard server-side ceiling for the avatar payload. Client pre-resizes to ~50KB. */ export const MAX_AVATAR_IMAGE_BYTES = 512 * 1024; +const avatarLog = createLogger("AgentAvatar"); + function isSupportedAvatarMime(mime: string): mime is SupportedAvatarImageMime { return SUPPORTED_AVATAR_IMAGE_MIMES.find((m) => m === mime) !== undefined; } /** - * Fetch the avatar image blob for an agent. Returns `null` when no image - * is set (the column is NULL). The data + mime pair is always coherent - * (set/cleared together by the service writes below). + * Fetch the avatar image descriptor for an agent. Returns `null` when no + * image is set. Exactly one payload location is populated: `objectKey` + * (object storage, key `avatars/`) for migrated/new avatars, or the + * legacy inline `data` bytea for rows the migration command has not moved + * yet. mime + updatedAt are always coherent with whichever is set. */ export async function getAgentAvatarImage( db: Database, uuid: string, -): Promise<{ data: Buffer; mime: string; updatedAt: Date } | null> { +): Promise<{ data: Buffer | null; objectKey: string | null; mime: string; updatedAt: Date } | null> { const [row] = await db .select({ data: agents.avatarImageData, + objectKey: agents.avatarObjectKey, mime: agents.avatarImageMime, updatedAt: agents.avatarImageUpdatedAt, }) .from(agents) .where(and(eq(agents.uuid, uuid), ne(agents.status, AGENT_STATUSES.DELETED))) .limit(1); - if (!row || !row.data || !row.mime || !row.updatedAt) return null; - return { data: row.data, mime: row.mime, updatedAt: row.updatedAt }; + if (!row || !row.mime || !row.updatedAt || (!row.data && !row.objectKey)) return null; + return { data: row.data, objectKey: row.objectKey, mime: row.mime, updatedAt: row.updatedAt }; } -/** Replace (or set) an agent's avatar image. Validates mime + size. */ -export async function setAgentAvatarImage(db: Database, uuid: string, data: Buffer, mime: string): Promise { - if (!isSupportedAvatarMime(mime)) { - throw new BadRequestError(`Unsupported avatar image type "${mime}". Use PNG, JPEG, or WEBP.`); +export type SetAgentAvatarImageOptions = { + mime: string; + /** Exact payload size from the request's Content-Length. */ + contentLength: number; +}; + +/** + * Replace (or set) an agent's avatar image. Validates mime + size up + * front (from the declared Content-Length), then STREAMS the payload to + * object storage under the deterministic `avatars/` key — overwrite + * in place, so replacements never accumulate objects — and records the + * key + mime + timestamp on the row. The legacy inline bytea is cleared on + * the way, so every avatar write converges rows toward the migrated shape. + */ +export async function setAgentAvatarImage( + db: Database, + objectStorage: ObjectStorage | null, + uuid: string, + body: Readable, + opts: SetAgentAvatarImageOptions, +): Promise { + if (!isSupportedAvatarMime(opts.mime)) { + throw new BadRequestError(`Unsupported avatar image type "${opts.mime}". Use PNG, JPEG, or WEBP.`); } - if (data.length === 0) { + if (opts.contentLength === 0) { throw new BadRequestError("Avatar image payload is empty."); } - if (data.length > MAX_AVATAR_IMAGE_BYTES) { - throw new BadRequestError(`Avatar image is too large (${data.length} bytes; max ${MAX_AVATAR_IMAGE_BYTES}).`); + if (opts.contentLength > MAX_AVATAR_IMAGE_BYTES) { + throw new BadRequestError( + `Avatar image is too large (${opts.contentLength} bytes; max ${MAX_AVATAR_IMAGE_BYTES}).`, + ); + } + if (!objectStorage) { + throw new ServiceUnavailableError( + "Object storage is not configured (FIRST_TREE_S3_*); avatar uploads are unavailable", + ); + } + const [existing] = await db + .select({ uuid: agents.uuid }) + .from(agents) + .where(and(eq(agents.uuid, uuid), ne(agents.status, AGENT_STATUSES.DELETED))) + .limit(1); + if (!existing) { + throw new NotFoundError(`Agent "${uuid}" not found`); } + + const key = avatarObjectKey(uuid); + const limiter = createByteLimitStream({ + expectedBytes: opts.contentLength, + makeMismatchError: (seenBytes) => + new BadRequestError( + `Avatar upload body does not match Content-Length (declared ${opts.contentLength}, saw ${seenBytes}${seenBytes > opts.contentLength ? "+" : ""} bytes)`, + ), + }); + await settleStreamingUpload({ + limiter, + producer: pipeline(body, limiter), + startConsumer: (abortSignal) => + objectStorage.putObjectStream(key, limiter, { + contentLength: opts.contentLength, + contentType: opts.mime, + abortSignal, + }), + }); + const now = new Date(); const result = await db .update(agents) .set({ - avatarImageData: data, - avatarImageMime: mime, + avatarObjectKey: key, + avatarImageData: null, + avatarImageMime: opts.mime, avatarImageUpdatedAt: now, updatedAt: now, }) .where(and(eq(agents.uuid, uuid), ne(agents.status, AGENT_STATUSES.DELETED))) .returning({ uuid: agents.uuid }); if (result.length === 0) { + // Agent vanished mid-upload; the just-written object is unreachable + // from any row, so remove it again (best-effort). + await objectStorage.deleteObject(key).catch(() => {}); throw new NotFoundError(`Agent "${uuid}" not found`); } return now; } -/** Clear an agent's avatar image (falls back to color + initial). */ -export async function clearAgentAvatarImage(db: Database, uuid: string): Promise { +/** + * Clear an agent's avatar image (falls back to color + initial). The row + * is cleared first so the UI converges immediately; the object delete is + * best-effort — a leftover object at the fixed key is unreachable and gets + * overwritten by the next upload. + */ +export async function clearAgentAvatarImage( + db: Database, + objectStorage: ObjectStorage | null, + uuid: string, +): Promise { const result = await db .update(agents) .set({ + avatarObjectKey: null, avatarImageData: null, avatarImageMime: null, avatarImageUpdatedAt: null, @@ -1374,6 +1459,12 @@ export async function clearAgentAvatarImage(db: Database, uuid: string): Promise if (result.length === 0) { throw new NotFoundError(`Agent "${uuid}" not found`); } + if (objectStorage) { + // Idempotent whether or not this avatar ever lived in object storage. + await objectStorage.deleteObject(avatarObjectKey(uuid)).catch((error) => { + avatarLog.warn({ err: error, uuid }, "avatar object delete failed; next upload overwrites the key"); + }); + } } /** diff --git a/packages/server/src/services/attachment-migration.ts b/packages/server/src/services/attachment-migration.ts new file mode 100644 index 000000000..8a96e4c97 --- /dev/null +++ b/packages/server/src/services/attachment-migration.ts @@ -0,0 +1,313 @@ +import { Readable } from "node:stream"; +import { and, asc, eq, gt, isNotNull, ne, sql } from "drizzle-orm"; +import type { Database } from "../db/connection.js"; +import { agents } from "../db/schema/agents.js"; +import { attachments } from "../db/schema/attachments.js"; +import { messages } from "../db/schema/messages.js"; +import { createLogger } from "../observability/logger.js"; +import { collectAttachmentIds } from "./attachment-references.js"; +import { attachmentObjectKey, avatarObjectKey, type ObjectStorage } from "./object-storage.js"; + +const log = createLogger("AttachmentMigration"); + +const MESSAGE_PAGE = 500; +const BLOB_BATCH = 8; +const BLOB_PARALLEL = 4; + +/** + * Test-only injection points (same pattern as the cron scheduler's + * `afterClaimForTest`): run between the object PUT and the row UPDATE so + * suites can deterministically exercise the mid-flight races the 0-row + * adjudication below exists for. Production callers pass nothing. + */ +export type AttachmentMigrationHooks = { + beforeAttachmentUpdate?: (attachmentId: string) => Promise; + beforeAvatarUpdate?: (agentUuid: string) => Promise; +}; + +export type AttachmentMigrationStats = { + organizationsBackfilled: number; + organizationlessRemaining: number; + messagesScanned: number; + edgesInserted: number; + attachmentsMoved: number; + attachmentsSkipped: number; + avatarsMoved: number; + avatarsSkipped: number; + attachmentsRemaining: number; + avatarsRemaining: number; +}; + +function affectedCount(result: unknown): number { + // postgres-js returns a RowList — an array carrying a `count` property + // with the command's affected-row count. + if (typeof result === "object" && result !== null && "count" in result) { + const count = (result as { count?: unknown }).count; + if (typeof count === "number") return count; + } + return 0; +} + +/** + * Move every inline binary payload (attachments.data, agents.avatar_image_data) + * into object storage and backfill governance metadata. Five idempotent + * phases — rerunning after a crash, or to converge references written + * concurrently with the message scan, is always safe: + * + * A. attachments.organization_id from the uploader's agent row + * B. attachment_references from message content/metadata (same + * `collectAttachmentIds` discovery the live write path uses; dangling + * historic ids and tombstoned rows are filtered by the join) + * C. attachment payloads → `attachments/` (atomic key+NULL swap; + * rows the sweep tombstoned mid-flight are skipped and the + * freshly-written object is removed again) + * D. avatar payloads → `avatars/` + * E. verify — counts of payloads still inline + * + * Runs against a live server: new uploads already land in object storage, + * the download path falls through to the deterministic key mid-migration, + * and the orphan sweep's verify scan keeps pre-backfill attachments alive + * until phase B records their edges. + */ +export async function migrateAttachmentsToObjectStorage( + db: Database, + storage: ObjectStorage, + hooks: AttachmentMigrationHooks = {}, +): Promise { + // ── Phase A: organization backfill ───────────────────────────────── + const orgBackfill = await db.execute(sql` + UPDATE attachments + SET organization_id = agents.organization_id + FROM agents + WHERE attachments.organization_id IS NULL + AND agents.uuid = attachments.uploaded_by + `); + const [orgless] = await db + .select({ count: sql`COUNT(*)` }) + .from(attachments) + .where(sql`${attachments.organizationId} IS NULL`); + const organizationsBackfilled = affectedCount(orgBackfill); + const organizationlessRemaining = Number(orgless?.count ?? 0); + log.info( + { organizationsBackfilled, organizationlessRemaining }, + "phase A: organization_id backfill (remaining NULL rows are quota-exempt legacy)", + ); + + // ── Phase B: reference-ledger backfill ───────────────────────────── + let edgesInserted = 0; + let messagesScanned = 0; + let cursor = ""; + for (;;) { + const page = await db + .select({ id: messages.id, content: messages.content, metadata: messages.metadata }) + .from(messages) + .where(gt(messages.id, cursor)) + .orderBy(asc(messages.id)) + .limit(MESSAGE_PAGE); + if (page.length === 0) break; + messagesScanned += page.length; + cursor = page[page.length - 1]?.id ?? cursor; + + const pairs: Array<{ attachment_id: string; message_id: string }> = []; + for (const row of page) { + for (const attachmentId of collectAttachmentIds(row.content, row.metadata)) { + pairs.push({ attachment_id: attachmentId, message_id: row.id }); + } + } + if (pairs.length === 0) continue; + try { + const inserted = await db.execute(sql` + INSERT INTO attachment_references (attachment_id, message_id) + SELECT p.attachment_id, p.message_id + FROM jsonb_to_recordset(${JSON.stringify(pairs)}::jsonb) + AS p(attachment_id text, message_id text) + JOIN attachments a ON a.id = p.attachment_id AND a.state != 'deleting' + ON CONFLICT DO NOTHING + `); + edgesInserted += affectedCount(inserted); + } catch (error) { + // Narrow FK race: an attachment can be destroyed between this + // statement's snapshot and its constraint checks. Skip the page and + // keep going — a rerun converges (the dangling target is gone by + // then, so its pair simply filters out). + log.warn({ err: error, page: cursor }, "phase B page failed; continuing (rerun converges)"); + } + } + log.info({ messagesScanned, edgesInserted }, "phase B: reference-ledger backfill"); + + // ── Phase C: attachment payloads → object storage ────────────────── + let attachmentsMoved = 0; + let attachmentsSkipped = 0; + for (;;) { + const batch = await db + .select({ + id: attachments.id, + mimeType: attachments.mimeType, + sizeBytes: attachments.sizeBytes, + data: attachments.data, + }) + .from(attachments) + .where(and(isNotNull(attachments.data), eq(attachments.state, "stored"))) + .orderBy(asc(attachments.id)) + .limit(BLOB_BATCH); + if (batch.length === 0) break; + + for (let i = 0; i < batch.length; i += BLOB_PARALLEL) { + await Promise.all( + batch.slice(i, i + BLOB_PARALLEL).map(async (row) => { + const data = row.data; + if (!data) return; + const key = attachmentObjectKey(row.id); + // Recheck right before the PUT: the batch read may be stale (a + // rival run migrated the row, or the sweep claimed it). Skipping + // here narrows the window in which we would overwrite the live + // object with these (identical-source) bytes for nothing. + const [fresh] = await db + .select({ state: attachments.state, hasData: isNotNull(attachments.data) }) + .from(attachments) + .where(eq(attachments.id, row.id)) + .limit(1); + if (!fresh || fresh.state !== "stored" || !fresh.hasData) { + attachmentsSkipped += 1; + return; + } + if (row.sizeBytes !== data.byteLength) { + log.warn( + { attachmentId: row.id, sizeBytes: row.sizeBytes, payloadBytes: data.byteLength }, + "size_bytes disagrees with payload; normalizing to the payload", + ); + } + await storage.putObjectStream(key, Readable.from([data]), { + contentLength: data.byteLength, + contentType: row.mimeType, + }); + await hooks.beforeAttachmentUpdate?.(row.id); + const updated = await db + .update(attachments) + .set({ objectKey: key, data: null, sizeBytes: data.byteLength }) + .where(and(eq(attachments.id, row.id), eq(attachments.state, "stored"), isNotNull(attachments.data))) + .returning({ id: attachments.id }); + if (updated.length === 0) { + // 0 rows has two very different causes — adjudicate by + // re-reading the row instead of deleting blindly: + // - row gone or tombstoned → the object is ownerless; delete it + // (otherwise it would leak forever — no row, so no sweep). + // - row alive with object_key set (a rival run won the swap) → + // the row OWNS this key; same key + same source bytes, so + // deleting here would destroy a live payload. Leave it. + const [current] = await db + .select({ state: attachments.state, objectKey: attachments.objectKey }) + .from(attachments) + .where(eq(attachments.id, row.id)) + .limit(1); + if (!current || current.state === "deleting" || !current.objectKey) { + await storage.deleteObject(key); + } + attachmentsSkipped += 1; + return; + } + attachmentsMoved += 1; + }), + ); + } + } + log.info({ attachmentsMoved, attachmentsSkipped }, "phase C: attachment payloads moved"); + + // ── Phase D: avatar payloads → object storage ────────────────────── + let avatarsMoved = 0; + let avatarsSkipped = 0; + for (;;) { + const batch = await db + .select({ uuid: agents.uuid, mime: agents.avatarImageMime, data: agents.avatarImageData }) + .from(agents) + .where(isNotNull(agents.avatarImageData)) + .orderBy(asc(agents.uuid)) + .limit(BLOB_BATCH); + if (batch.length === 0) break; + + for (const row of batch) { + const data = row.data; + if (!data) continue; + if (!row.mime) { + // mime is NULL iff data is NULL by contract; a violating row cannot + // be served today either — clear it rather than migrating garbage. + log.warn({ agentUuid: row.uuid }, "avatar payload without mime; clearing without migrating"); + await db + .update(agents) + .set({ avatarImageData: null }) + .where(and(eq(agents.uuid, row.uuid), isNotNull(agents.avatarImageData))); + avatarsSkipped += 1; + continue; + } + const key = avatarObjectKey(row.uuid); + // Recheck right before the PUT: an online avatar upload may have + // landed since the batch read (new payload already at this key, row + // bytea cleared). Skipping avoids overwriting the fresh upload with + // these older bytes — the fixed per-agent key is last-writer-wins, + // so this recheck is what keeps the race window negligible. + const [freshAgent] = await db + .select({ hasData: isNotNull(agents.avatarImageData) }) + .from(agents) + .where(eq(agents.uuid, row.uuid)) + .limit(1); + if (!freshAgent || !freshAgent.hasData) { + avatarsSkipped += 1; + continue; + } + await storage.putObjectStream(key, Readable.from([data]), { + contentLength: data.byteLength, + contentType: row.mime, + }); + await hooks.beforeAvatarUpdate?.(row.uuid); + const updated = await db + .update(agents) + .set({ avatarObjectKey: key, avatarImageData: null }) + .where(and(eq(agents.uuid, row.uuid), isNotNull(agents.avatarImageData))) + .returning({ uuid: agents.uuid }); + if (updated.length === 0) { + // Adjudicate like phase C: only delete the object when no live row + // claims the key. An online upload that raced us has already set + // avatar_object_key — deleting here would 404 the avatar the user + // just uploaded. + const [current] = await db + .select({ objectKey: agents.avatarObjectKey }) + .from(agents) + .where(eq(agents.uuid, row.uuid)) + .limit(1); + if (!current || !current.objectKey) { + await storage.deleteObject(key); + } + avatarsSkipped += 1; + } else { + avatarsMoved += 1; + } + } + } + log.info({ avatarsMoved, avatarsSkipped }, "phase D: avatar payloads moved"); + + // ── Phase E: verify ──────────────────────────────────────────────── + const [attachmentsLeft] = await db + .select({ count: sql`COUNT(*)` }) + .from(attachments) + .where(and(isNotNull(attachments.data), ne(attachments.state, "deleting"))); + const [avatarsLeft] = await db + .select({ count: sql`COUNT(*)` }) + .from(agents) + .where(isNotNull(agents.avatarImageData)); + const attachmentsRemaining = Number(attachmentsLeft?.count ?? 0); + const avatarsRemaining = Number(avatarsLeft?.count ?? 0); + log.info({ attachmentsRemaining, avatarsRemaining }, "phase E: inline payloads remaining"); + + return { + organizationsBackfilled, + organizationlessRemaining, + messagesScanned, + edgesInserted, + attachmentsMoved, + attachmentsSkipped, + avatarsMoved, + avatarsSkipped, + attachmentsRemaining, + avatarsRemaining, + }; +} diff --git a/packages/server/src/services/attachment-references.ts b/packages/server/src/services/attachment-references.ts new file mode 100644 index 000000000..35616f25c --- /dev/null +++ b/packages/server/src/services/attachment-references.ts @@ -0,0 +1,195 @@ +import { attachmentRefsFromMetadata, isImageBatchRefContent, isImageRefContent } from "@first-tree/shared"; +import { and, eq, inArray } from "drizzle-orm"; +import type { Database } from "../db/connection.js"; +import { attachmentReferences } from "../db/schema/attachment-references.js"; +import { attachments } from "../db/schema/attachments.js"; +import { BadRequestError } from "../errors.js"; +import { createLogger } from "../observability/logger.js"; +import type { ObjectStorage } from "./object-storage.js"; + +const log = createLogger("AttachmentReferences"); + +/** `Database` or an open transaction — both satisfy the query surface used here. */ +type ReferenceWriter = Pick; + +function asMetadataRecord(value: unknown): Record | undefined { + if (typeof value !== "object" || value === null || Array.isArray(value)) return undefined; + // Structural narrow only — jsonb columns surface as `unknown`. + return value as Record; +} + +/** + * THE single source of truth for attachment-reference discovery on a + * message. Every shape that can carry an `attachments.id` lives here: + * + * - `content` single image ref (`{ imageId, ... }`, format "file") + * - `content` batch ref (`{ attachments: [{ imageId, ... }] }`, format "file") + * - `metadata.attachments[]` generic refs (`{ attachmentId, ... }`) + * + * Reference writes (send/edit), the migration backfill, and the orphan + * sweep's verify scan all derive from this function — a future reference + * shape must be added here and nowhere else. Agent avatars are NOT + * attachment references (separate storage namespace; see db/schema/agents.ts). + */ +export function collectAttachmentIds(content: unknown, metadata: unknown): Set { + const ids = new Set(); + if (isImageRefContent(content)) { + ids.add(content.imageId); + } else if (isImageBatchRefContent(content)) { + for (const image of content.attachments) { + ids.add(image.imageId); + } + } + for (const ref of attachmentRefsFromMetadata(asMetadataRecord(metadata))) { + ids.add(ref.attachmentId); + } + return ids; +} + +export type SyncMessageAttachmentReferencesInput = { + messageId: string; + /** Organization owning the chat the message lives in. */ + organizationId: string; + content: unknown; + metadata: unknown; +}; + +export type SyncMessageAttachmentReferencesResult = { + /** + * Attachments this sync CAS-ed to `deleting` because the message dropped + * their last remaining reference. The caller destroys them after commit + * via `destroyDeletingAttachments` (the sweep is the crash backstop). + */ + removedForDeletion: string[]; +}; + +/** + * Reconcile the `attachment_references` ledger with what a message + * actually references — called INSIDE the message write transaction, right + * after the row insert/update, for both send (no prior edges) and edit. + * + * Locking protocol: one `SELECT ... ORDER BY id FOR UPDATE` over the whole + * touched set (added ∪ removed). Every reference/deletion path locks + * attachment rows in ascending-id order in a single statement, so lock + * acquisition forms no cycles; the sweep's SKIP LOCKED passes never wait. + * After locking, added references are validated (row exists, `stored`, + * same-org or legacy NULL org) — a `pending` upload or `deleting` + * tombstone cannot gain references, which is what makes the tombstone CAS + * race-free in the other direction. + */ +export async function syncMessageAttachmentReferences( + tx: ReferenceWriter, + input: SyncMessageAttachmentReferencesInput, +): Promise { + const target = collectAttachmentIds(input.content, input.metadata); + const currentRows = await tx + .select({ attachmentId: attachmentReferences.attachmentId }) + .from(attachmentReferences) + .where(eq(attachmentReferences.messageId, input.messageId)); + const current = new Set(currentRows.map((row) => row.attachmentId)); + + const added = [...target].filter((id) => !current.has(id)); + const removed = [...current].filter((id) => !target.has(id)); + if (added.length === 0 && removed.length === 0) { + return { removedForDeletion: [] }; + } + + const touched = [...new Set([...added, ...removed])].sort(); + const lockedRows = await tx + .select({ + id: attachments.id, + state: attachments.state, + organizationId: attachments.organizationId, + }) + .from(attachments) + .where(inArray(attachments.id, touched)) + .orderBy(attachments.id) + .for("update"); + const lockedById = new Map(lockedRows.map((row) => [row.id, row])); + + for (const id of added) { + const row = lockedById.get(id); + if (!row) { + throw new BadRequestError(`Message references unknown attachment "${id}"`); + } + if (row.state !== "stored") { + // `pending` = still uploading, `deleting` = tombstoned — neither is + // referenceable, and rejecting here closes the reference-vs-delete race. + throw new BadRequestError(`Message references attachment "${id}" which is not available`); + } + if (row.organizationId !== null && row.organizationId !== input.organizationId) { + throw new BadRequestError(`Message references attachment "${id}" from a different organization`); + } + } + + if (added.length > 0) { + await tx + .insert(attachmentReferences) + .values(added.map((attachmentId) => ({ attachmentId, messageId: input.messageId }))) + .onConflictDoNothing(); + } + + const removedForDeletion: string[] = []; + if (removed.length > 0) { + await tx + .delete(attachmentReferences) + .where( + and(eq(attachmentReferences.messageId, input.messageId), inArray(attachmentReferences.attachmentId, removed)), + ); + // Which of the removed attachments now have zero edges? Their rows are + // locked above, so no concurrent send can add an edge until we commit. + const stillReferenced = new Set( + ( + await tx + .selectDistinct({ attachmentId: attachmentReferences.attachmentId }) + .from(attachmentReferences) + .where(inArray(attachmentReferences.attachmentId, removed)) + ).map((row) => row.attachmentId), + ); + const orphaned = removed.filter((id) => !stillReferenced.has(id) && lockedById.get(id)?.state === "stored"); + if (orphaned.length > 0) { + const tombstoned = await tx + .update(attachments) + .set({ state: "deleting" }) + .where(and(inArray(attachments.id, orphaned), eq(attachments.state, "stored"))) + .returning({ id: attachments.id }); + removedForDeletion.push(...tombstoned.map((row) => row.id)); + } + } + + return { removedForDeletion }; +} + +/** + * Destroy `deleting` tombstones: remove the payload object, then the row. + * Best-effort and idempotent — any failure leaves the tombstone for the + * background sweep's retry pass. Rows whose payload never reached object + * storage (legacy bytea) skip the object delete; when object storage is + * unconfigured the tombstone is left in place so the object is not leaked. + */ +export async function destroyDeletingAttachments( + db: Database, + objectStorage: ObjectStorage | null, + ids: readonly string[], +): Promise { + for (const id of ids) { + try { + const [row] = await db + .select({ objectKey: attachments.objectKey }) + .from(attachments) + .where(and(eq(attachments.id, id), eq(attachments.state, "deleting"))) + .limit(1); + if (!row) continue; + if (row.objectKey) { + if (!objectStorage) { + log.warn({ attachmentId: id }, "object storage unavailable; leaving tombstone for the sweep"); + continue; + } + await objectStorage.deleteObject(row.objectKey); + } + await db.delete(attachments).where(and(eq(attachments.id, id), eq(attachments.state, "deleting"))); + } catch (error) { + log.warn({ err: error, attachmentId: id }, "attachment destroy failed; the sweep will retry"); + } + } +} diff --git a/packages/server/src/services/attachment-sweep.ts b/packages/server/src/services/attachment-sweep.ts new file mode 100644 index 000000000..2d6a32e07 --- /dev/null +++ b/packages/server/src/services/attachment-sweep.ts @@ -0,0 +1,162 @@ +import { and, eq, inArray, lt, notExists, sql } from "drizzle-orm"; +import type { Database } from "../db/connection.js"; +import { attachmentReferences } from "../db/schema/attachment-references.js"; +import { attachments } from "../db/schema/attachments.js"; +import { createLogger } from "../observability/logger.js"; +import { destroyDeletingAttachments } from "./attachment-references.js"; +import type { ObjectStorage } from "./object-storage.js"; + +const log = createLogger("AttachmentSweep"); + +export type AttachmentSweepOptions = { + /** Age after which an unreferenced `stored` attachment is an orphan (governed default 24h). */ + orphanGraceSeconds: number; + /** Age after which a `pending` reservation whose upload never finalized is reclaimed. */ + pendingTtlSeconds: number; + /** Per-pass claim cap; bounds lock hold time and the verify-scan candidate set. */ + batchSize?: number; +}; + +export type AttachmentSweepResult = { + pendingReclaimed: number; + orphansDeleted: number; + /** Orphan candidates saved by the verify scan (referenced in message text but missing edges). */ + orphansVetoed: number; + tombstonesCleared: number; +}; + +/** + * Orphan / reservation / tombstone sweep. Runs concurrently on every + * replica with no leader election: every pass claims its batch with + * `FOR UPDATE SKIP LOCKED` (the same claim shape as the cron scheduler), + * transitions are CAS-guarded, and object deletion is idempotent — two + * replicas sweeping at once just split the batch. + * + * Three passes per run: + * + * 1. expired `pending` reservations → tombstone → destroy + * 2. aged zero-edge `stored` rows → VERIFY SCAN → tombstone → destroy + * 3. leftover `deleting` tombstones → destroy (crash retry) + * + * The verify scan is the load-bearing safety net, not an optimization: in + * the window between deploying this feature and running the + * `migrate:attachments` backfill, EVERY pre-existing attachment has zero + * edges — deleting on the edge ledger alone would destroy referenced + * data. Before tombstoning, candidates are matched as literal text against + * `messages.content`/`metadata` (one scan for the whole batch via + * `unnest`); any hit is vetoed and logged. A real reference always + * contains the id verbatim in one of those two jsonb columns, so the scan + * can only err toward keeping (false positives keep an orphan alive until + * the backfill records real edges; false negatives cannot happen). + */ +export async function sweepAttachments( + db: Database, + objectStorage: ObjectStorage | null, + opts: AttachmentSweepOptions, +): Promise { + const batchSize = opts.batchSize ?? 50; + const result: AttachmentSweepResult = { + pendingReclaimed: 0, + orphansDeleted: 0, + orphansVetoed: 0, + tombstonesCleared: 0, + }; + + // Pass 1 — expired pending reservations (crashed/abandoned uploads). + const pendingCutoff = new Date(Date.now() - opts.pendingTtlSeconds * 1000); + const expiredPending = await db.transaction(async (tx) => { + const claimed = await tx + .select({ id: attachments.id }) + .from(attachments) + .where(and(eq(attachments.state, "pending"), lt(attachments.createdAt, pendingCutoff))) + .orderBy(attachments.createdAt) + .limit(batchSize) + .for("update", { skipLocked: true }); + const ids = claimed.map((row) => row.id); + if (ids.length > 0) { + await tx.update(attachments).set({ state: "deleting" }).where(inArray(attachments.id, ids)); + } + return ids; + }); + await destroyDeletingAttachments(db, objectStorage, expiredPending); + result.pendingReclaimed = expiredPending.length; + + // Pass 2 — aged zero-edge stored rows, with the verify-scan veto. + const orphanCutoff = new Date(Date.now() - opts.orphanGraceSeconds * 1000); + const orphans = await db.transaction(async (tx) => { + const candidates = await tx + .select({ id: attachments.id }) + .from(attachments) + .where( + and( + eq(attachments.state, "stored"), + lt(attachments.createdAt, orphanCutoff), + notExists( + tx + .select({ one: sql`1` }) + .from(attachmentReferences) + .where(eq(attachmentReferences.attachmentId, attachments.id)), + ), + ), + ) + .orderBy(attachments.createdAt) + .limit(batchSize) + .for("update", { skipLocked: true }); + const candidateIds = candidates.map((row) => row.id); + if (candidateIds.length === 0) { + return { deleted: [] as string[], vetoed: 0 }; + } + + // One scan of `messages` for the whole batch: an id referenced anywhere + // in content/metadata jsonb appears as a literal substring of its text + // form. Candidate attachment rows are locked above, so a concurrent + // send targeting one of them blocks until this transaction commits. + const vetoRows = await tx.execute(sql` + SELECT DISTINCT c.id + FROM jsonb_array_elements_text(${JSON.stringify(candidateIds)}::jsonb) AS c(id) + JOIN messages m + ON m.content::text LIKE '%' || c.id || '%' + OR m.metadata::text LIKE '%' || c.id || '%' + `); + const vetoed = new Set(); + for (const row of vetoRows) { + if (typeof row === "object" && row !== null && "id" in row && typeof row.id === "string") { + vetoed.add(row.id); + } + } + if (vetoed.size > 0) { + log.warn( + { attachmentIds: [...vetoed] }, + "orphan sweep vetoed candidates referenced in message text without ledger edges (run migrate:attachments to backfill)", + ); + } + const survivors = candidateIds.filter((id) => !vetoed.has(id)); + if (survivors.length > 0) { + await tx + .update(attachments) + .set({ state: "deleting" }) + .where(and(inArray(attachments.id, survivors), eq(attachments.state, "stored"))); + } + return { deleted: survivors, vetoed: vetoed.size }; + }); + await destroyDeletingAttachments(db, objectStorage, orphans.deleted); + result.orphansDeleted = orphans.deleted.length; + result.orphansVetoed = orphans.vetoed; + + // Pass 3 — leftover tombstones (crashes between CAS and destroy, or + // storage that was unavailable on an earlier attempt). + const tombstones = await db.transaction(async (tx) => { + const claimed = await tx + .select({ id: attachments.id }) + .from(attachments) + .where(eq(attachments.state, "deleting")) + .orderBy(attachments.createdAt) + .limit(batchSize) + .for("update", { skipLocked: true }); + return claimed.map((row) => row.id); + }); + await destroyDeletingAttachments(db, objectStorage, tombstones); + result.tombstonesCleared = tombstones.length; + + return result; +} diff --git a/packages/server/src/services/attachment.ts b/packages/server/src/services/attachment.ts index caf997084..a18e46b25 100644 --- a/packages/server/src/services/attachment.ts +++ b/packages/server/src/services/attachment.ts @@ -1,46 +1,74 @@ import { randomUUID } from "node:crypto"; -import { MAX_ATTACHMENT_BYTES } from "@first-tree/shared"; -import { eq } from "drizzle-orm"; +import { ATTACHMENT_ERROR_CODES, MAX_ATTACHMENT_BYTES } from "@first-tree/shared"; +import { and, eq, inArray, sql } from "drizzle-orm"; import type { Database } from "../db/connection.js"; import { attachments } from "../db/schema/attachments.js"; -import { BadRequestError } from "../errors.js"; +import { BadRequestError, UnprocessableError } from "../errors.js"; +import { attachmentObjectKey } from "./object-storage.js"; /** - * Object-storage primitive (M1). + * Attachment metadata service. * - * - `createAttachment` writes a row, returns the inserted record. - * - `loadAttachmentMeta` looks up by id WITHOUT the `bytea` payload; returns - * `null` on miss so the route layer can emit a 404 cleanly. The download - * route runs the ETag check off the metadata so a cache hit (304) never - * pulls the blob out of PG. - * - `loadAttachmentData` fetches just the `bytea` payload, called only once - * the route has decided to actually stream bytes. + * Upload is reservation-first (see `reserveAttachment`): the row is inserted + * in state `pending` BEFORE any byte reaches object storage, which makes the + * quota reservation durable, then flipped to `stored` once the payload is + * verified (`finalizeAttachment`). Reads: * - * Download authorization is a capability model handled at the route layer: - * a valid user JWT plus knowledge of the unguessable id. The service holds no - * ACL logic. + * - `loadAttachmentMeta` looks up by id WITHOUT the legacy `bytea` payload; + * returns `null` on miss so the route layer can emit a 404 cleanly. The + * download route runs the ETag check off the metadata so a cache hit + * (304) never touches the payload. + * - `loadAttachmentData` fetches just the legacy `bytea` payload — only + * used for rows the migration command has not moved to object storage. * - * Service throws `BadRequestError` for input validation (oversize / empty - * bytes / blank mime). Route layer maps service exceptions to HTTP. + * Download authorization is a capability model handled at the route layer: + * a valid user JWT plus knowledge of the unguessable id. The service holds + * no ACL logic. */ export type AttachmentRow = typeof attachments.$inferSelect; -export type CreateAttachmentInput = { - /** Optional caller-supplied id (UUIDv4). Generated when absent. */ - id?: string; +/** Everything in `AttachmentRow` except the legacy `bytea` payload. */ +export type AttachmentMeta = Omit; + +/** A select-capable executor — `Database` or a transaction both satisfy it, so + * read helpers can run inside a caller's open transaction. */ +export type AttachmentReader = Pick; + +export type OrgAttachmentQuota = { + maxTotalBytes: number; + maxObjectCount: number; +}; + +export type ReserveAttachmentInput = { + organizationId: string; mimeType: string; filename: string; - data: Buffer; + /** Exact payload size from the request's Content-Length. */ + sizeBytes: number; /** `agents.uuid` of the uploader; humans pass their humanAgentId. */ uploadedBy: string; + quota: OrgAttachmentQuota; }; -export async function createAttachment(db: Database, input: CreateAttachmentInput): Promise { - if (input.data.byteLength === 0) { +/** + * Reserve quota and create the `pending` attachment row. + * + * Runs in one transaction holding the per-org advisory xact lock (same + * two-int form as the landing-campaign quota locks), so concurrent uploads + * of one org serialize on admission and cannot jointly overshoot the + * quota. Release paths (sweep, delete) take no lock: a concurrent decrease + * can only make this check conservative, never over-admit. + * + * Throws `UnprocessableError` with the stable `ATTACHMENT_QUOTA_EXCEEDED` + * wire code when either the org byte quota or object-count quota would be + * exceeded. Input validation errors are `BadRequestError`. + */ +export async function reserveAttachment(db: Database, input: ReserveAttachmentInput): Promise { + if (input.sizeBytes <= 0) { throw new BadRequestError("Attachment is empty"); } - if (input.data.byteLength > MAX_ATTACHMENT_BYTES) { + if (input.sizeBytes > MAX_ATTACHMENT_BYTES) { throw new BadRequestError(`Attachment exceeds maximum size of ${MAX_ATTACHMENT_BYTES} bytes`); } if (input.mimeType.trim().length === 0) { @@ -50,41 +78,100 @@ export async function createAttachment(db: Database, input: CreateAttachmentInpu throw new BadRequestError("Attachment filename is required"); } - const id = input.id ?? randomUUID(); - const [row] = await db - .insert(attachments) - .values({ - id, - mimeType: input.mimeType, - filename: input.filename, - sizeBytes: input.data.byteLength, - data: input.data, - uploadedBy: input.uploadedBy, - }) - .returning(); - if (!row) { - // Drizzle returns the inserted row(s); empty array would only happen on - // a driver bug. Throw rather than swallow — caller would see a wrong- - // shape return otherwise. - throw new Error("Attachment insert returned no row"); - } - return row; + const id = randomUUID(); + return await db.transaction(async (tx) => { + await tx.execute( + sql`SELECT pg_advisory_xact_lock(hashtext('attachment_quota'), hashtext(${input.organizationId}))`, + ); + const [usage] = await tx + .select({ + totalBytes: sql`COALESCE(SUM(${attachments.sizeBytes}), 0)`, + objectCount: sql`COUNT(*)`, + }) + .from(attachments) + .where( + and(eq(attachments.organizationId, input.organizationId), inArray(attachments.state, ["pending", "stored"])), + ); + const totalBytes = Number(usage?.totalBytes ?? 0); + const objectCount = Number(usage?.objectCount ?? 0); + if (totalBytes + input.sizeBytes > input.quota.maxTotalBytes) { + throw new UnprocessableError( + `Organization attachment storage quota exceeded (${totalBytes} of ${input.quota.maxTotalBytes} bytes used; upload is ${input.sizeBytes} bytes)`, + { code: ATTACHMENT_ERROR_CODES.quotaExceeded, "attachment.quota.dimension": "bytes" }, + ); + } + if (objectCount + 1 > input.quota.maxObjectCount) { + throw new UnprocessableError( + `Organization attachment count quota exceeded (${objectCount} of ${input.quota.maxObjectCount} objects used)`, + { code: ATTACHMENT_ERROR_CODES.quotaExceeded, "attachment.quota.dimension": "count" }, + ); + } + + const [row] = await tx + .insert(attachments) + .values({ + id, + organizationId: input.organizationId, + mimeType: input.mimeType, + filename: input.filename, + sizeBytes: input.sizeBytes, + objectKey: attachmentObjectKey(id), + state: "pending", + data: null, + uploadedBy: input.uploadedBy, + }) + .returning({ + id: attachments.id, + organizationId: attachments.organizationId, + mimeType: attachments.mimeType, + filename: attachments.filename, + sizeBytes: attachments.sizeBytes, + objectKey: attachments.objectKey, + state: attachments.state, + uploadedBy: attachments.uploadedBy, + createdAt: attachments.createdAt, + }); + if (!row) { + throw new Error("Attachment reservation insert returned no row"); + } + return row; + }); } -/** Everything in `AttachmentRow` except the `bytea` payload. */ -export type AttachmentMeta = Omit; +/** + * CAS the reservation to `stored` after the payload is verified in object + * storage. Returns `false` when the row is no longer `pending` — i.e. the + * upload outlived the pending TTL and the sweep reclaimed the reservation; + * the caller must then delete the freshly-written object. + */ +export async function finalizeAttachment(db: Database, id: string): Promise { + const rows = await db + .update(attachments) + .set({ state: "stored" }) + .where(and(eq(attachments.id, id), eq(attachments.state, "pending"))) + .returning({ id: attachments.id }); + return rows.length > 0; +} -/** A select-capable executor — `Database` or a transaction both satisfy it, so - * read helpers can run inside a caller's open transaction. */ -export type AttachmentReader = Pick; +/** + * Drop a `pending` reservation after a failed upload (stream error, client + * abort, storage failure). Best-effort: if the process dies before this + * runs, the pending-TTL sweep reclaims the row instead. + */ +export async function deletePendingReservation(db: Database, id: string): Promise { + await db.delete(attachments).where(and(eq(attachments.id, id), eq(attachments.state, "pending"))); +} export async function loadAttachmentMeta(db: AttachmentReader, id: string): Promise { const [row] = await db .select({ id: attachments.id, + organizationId: attachments.organizationId, mimeType: attachments.mimeType, filename: attachments.filename, sizeBytes: attachments.sizeBytes, + objectKey: attachments.objectKey, + state: attachments.state, uploadedBy: attachments.uploadedBy, createdAt: attachments.createdAt, }) @@ -98,3 +185,59 @@ export async function loadAttachmentData(db: Database, id: string): Promise { + if (input.data.byteLength === 0) { + throw new BadRequestError("Attachment is empty"); + } + if (input.data.byteLength > MAX_ATTACHMENT_BYTES) { + throw new BadRequestError(`Attachment exceeds maximum size of ${MAX_ATTACHMENT_BYTES} bytes`); + } + const id = input.id ?? randomUUID(); + const [row] = await db + .insert(attachments) + .values({ + id, + organizationId: null, + mimeType: input.mimeType, + filename: input.filename, + sizeBytes: input.data.byteLength, + objectKey: null, + state: "stored", + data: input.data, + uploadedBy: input.uploadedBy, + }) + .returning({ + id: attachments.id, + organizationId: attachments.organizationId, + mimeType: attachments.mimeType, + filename: attachments.filename, + sizeBytes: attachments.sizeBytes, + objectKey: attachments.objectKey, + state: attachments.state, + uploadedBy: attachments.uploadedBy, + createdAt: attachments.createdAt, + }); + if (!row) { + throw new Error("Attachment insert returned no row"); + } + return row; +} diff --git a/packages/server/src/services/background-tasks.ts b/packages/server/src/services/background-tasks.ts index 76a649e0a..d6d944add 100644 --- a/packages/server/src/services/background-tasks.ts +++ b/packages/server/src/services/background-tasks.ts @@ -1,5 +1,6 @@ import type { FastifyInstance } from "fastify"; import { createLogger } from "../observability/index.js"; +import { sweepAttachments } from "./attachment-sweep.js"; import * as chatArchiveService from "./chat-archive.js"; import * as clientService from "./client.js"; import { createCronScheduler } from "./cron-scheduler.js"; @@ -18,6 +19,8 @@ export function createBackgroundTasks(app: FastifyInstance, instanceId: string): let inboxTimer: ReturnType | null = null; let heartbeatTimer: ReturnType | null = null; let archiveSweepTimer: ReturnType | null = null; + let attachmentSweepTimer: ReturnType | null = null; + let attachmentSweepRunning = false; const cronScheduler = createCronScheduler(app); return { @@ -70,6 +73,35 @@ export function createBackgroundTasks(app: FastifyInstance, instanceId: string): }, archiveSweepSeconds * 1000); } + const attachmentSweepSeconds = app.config.attachments.sweepIntervalSeconds; + if (attachmentSweepSeconds > 0) { + attachmentSweepTimer = setInterval(async () => { + // Per-instance overlap latch only — cross-replica concurrency is + // safe by construction (SKIP LOCKED claims + idempotent destroy), + // but stacking runs on one slow instance would just add load. + if (attachmentSweepRunning) return; + attachmentSweepRunning = true; + try { + const stats = await sweepAttachments(app.db, app.objectStorage, { + orphanGraceSeconds: app.config.attachments.orphanGraceSeconds, + pendingTtlSeconds: app.config.attachments.pendingTtlSeconds, + }); + if ( + stats.pendingReclaimed > 0 || + stats.orphansDeleted > 0 || + stats.orphansVetoed > 0 || + stats.tombstonesCleared > 0 + ) { + log.info(stats, "attachment sweep pass completed"); + } + } catch (err) { + log.error({ err }, "attachment sweep failed"); + } finally { + attachmentSweepRunning = false; + } + }, attachmentSweepSeconds * 1000); + } + cronScheduler.start(); presenceService.heartbeatInstance(app.db, instanceId).catch((err) => { @@ -91,6 +123,10 @@ export function createBackgroundTasks(app: FastifyInstance, instanceId: string): clearInterval(archiveSweepTimer); archiveSweepTimer = null; } + if (attachmentSweepTimer) { + clearInterval(attachmentSweepTimer); + attachmentSweepTimer = null; + } }, }; } diff --git a/packages/server/src/services/message.ts b/packages/server/src/services/message.ts index aadd7f04a..b6ad416c6 100644 --- a/packages/server/src/services/message.ts +++ b/packages/server/src/services/message.ts @@ -27,10 +27,12 @@ import { BadRequestError, ForbiddenError, NotFoundError } from "../errors.js"; import { createLogger, messageAttrs, withSpan } from "../observability/index.js"; import { uuidv7 } from "../uuid.js"; import { upsertSessionState } from "./activity.js"; +import { destroyDeletingAttachments, syncMessageAttachmentReferences } from "./attachment-references.js"; import { applyAfterFanOut, fireChatMessageKick } from "./chat-projection.js"; import { validateDocumentContext, validateMessageAttachmentRefs } from "./doc-snapshots.js"; import { hasRemainingLandingCampaignTrialBudget } from "./landing-campaigns/chat-state.js"; import { getLandingCampaignTrialChat, withLandingCampaignChatState } from "./landing-campaigns/metadata.js"; +import type { ObjectStorage } from "./object-storage.js"; const log = createLogger("message"); const ADDRESSED_AGENT_IDS_METADATA_KEY = "addressedAgentIds"; @@ -688,11 +690,18 @@ async function sendMessageInner( .from(agents) .where(eq(agents.uuid, senderId)) .limit(1), - tx.select({ metadata: chats.metadata }).from(chats).where(eq(chats.id, chatId)).limit(1), + tx + .select({ metadata: chats.metadata, organizationId: chats.organizationId }) + .from(chats) + .where(eq(chats.id, chatId)) + .limit(1), ]); if (!senderRow) { throw new NotFoundError(`Sender agent "${senderId}" not found`); } + if (!chatRowSnapshot) { + throw new NotFoundError(`Chat "${chatId}" not found`); + } const initialTrial = getLandingCampaignTrialChat(chatRowSnapshot); // Trial chat state is a server-owned single-run state machine. Lock and // re-read only those rows so concurrent outbox writes cannot apply stale @@ -753,6 +762,17 @@ async function sendMessageInner( }) .returning(); + // 3b. Record attachment references (content imageIds + metadata refs) + // in the same transaction, validating each referenced attachment is + // a live same-org `stored` row — the first existence gate content + // imageIds ever get, and the write side of the orphan-sweep ledger. + await syncMessageAttachmentReferences(tx, { + messageId, + organizationId: chatRowSnapshot.organizationId, + content: outboundContent, + metadata: metadataToStore, + }); + // 4. Fan-out: create inbox entries for every non-sender participant. // The `notify` flag splits them in two: // - `notify=true` — wakes the recipient's session (the existing path). @@ -1051,67 +1071,101 @@ export function maybeUnwrapDoubleEncoded(content: string): string | null { export async function editMessage( db: Database, + objectStorage: ObjectStorage | null, chatId: string, messageId: string, senderId: string, data: { format?: string; content?: unknown }, ) { - const [msg] = await db.select().from(messages).where(eq(messages.id, messageId)).limit(1); - if (!msg) throw new NotFoundError(`Message "${messageId}" not found`); - if (msg.chatId !== chatId) throw new NotFoundError(`Message "${messageId}" not found in this chat`); - if (msg.senderId !== senderId) throw new ForbiddenError("Only the sender can edit a message"); - const protectedContextReviewKey = Object.keys(msg.metadata).find( - (key) => key === "contextTreeReviewer" || key.startsWith("contextReview"), - ); - if (protectedContextReviewKey) { - throw new ForbiddenError("Context Reviewer run history cannot be edited"); - } + // Transactional: the content swap and the attachment-reference ledger + // reconciliation must land atomically — a content edit is the one live + // event that can DROP a reference, and dropping the last one tombstones + // the attachment inside the same transaction. + const { updated, removedForDeletion } = await db.transaction(async (tx) => { + const [msg] = await tx.select().from(messages).where(eq(messages.id, messageId)).limit(1); + if (!msg) throw new NotFoundError(`Message "${messageId}" not found`); + if (msg.chatId !== chatId) throw new NotFoundError(`Message "${messageId}" not found in this chat`); + if (msg.senderId !== senderId) throw new ForbiddenError("Only the sender can edit a message"); + const protectedContextReviewKey = Object.keys(msg.metadata).find( + (key) => key === "contextTreeReviewer" || key.startsWith("contextReview"), + ); + if (protectedContextReviewKey) { + throw new ForbiddenError("Context Reviewer run history cannot be edited"); + } - // The open-question counter (`open_request_count`) is maintained only on the - // send path, keyed off `format=request`. Allowing an edit to flip a message - // into or out of `request` would desync that counter (a request edited to - // text leaves a stuck +1; text edited to request renders an open card with - // no count). Forbid format changes that touch `request`; content edits and - // other format changes are unaffected. See proposals/group-chat-unified-send §D1. - if ( - data.format !== undefined && - data.format !== msg.format && - (data.format === MESSAGE_FORMATS.REQUEST || msg.format === MESSAGE_FORMATS.REQUEST) - ) { - throw new BadRequestError("Cannot change a message's format to or from 'request'."); - } + // The open-question counter (`open_request_count`) is maintained only on the + // send path, keyed off `format=request`. Allowing an edit to flip a message + // into or out of `request` would desync that counter (a request edited to + // text leaves a stuck +1; text edited to request renders an open card with + // no count). Forbid format changes that touch `request`; content edits and + // other format changes are unaffected. See proposals/group-chat-unified-send §D1. + if ( + data.format !== undefined && + data.format !== msg.format && + (data.format === MESSAGE_FORMATS.REQUEST || msg.format === MESSAGE_FORMATS.REQUEST) + ) { + throw new BadRequestError("Cannot change a message's format to or from 'request'."); + } - const setClause: Record = {}; - if (data.format !== undefined) setClause.format = data.format; - if (data.content !== undefined) { - // An edit can replace the body of any message — including an already-open - // `format=request` ask whose format is frozen above. Reuse the send-path - // guards against the effective post-edit `{ format, content }` so an edit - // can't turn a live message into an empty / placeholder blocking card or an - // agent-authored escaped-newline body. - const [senderRow] = await db.select({ type: agents.type }).from(agents).where(eq(agents.uuid, senderId)).limit(1); - if (!senderRow) throw new NotFoundError(`Sender agent "${senderId}" not found`); - const effectiveContent = normalizeNonHumanTextContent({ - chatId, - senderId, - senderType: senderRow.type, - content: data.content, + const setClause: Record = {}; + if (data.format !== undefined) setClause.format = data.format; + if (data.content !== undefined) { + // An edit can replace the body of any message — including an already-open + // `format=request` ask whose format is frozen above. Reuse the send-path + // guards against the effective post-edit `{ format, content }` so an edit + // can't turn a live message into an empty / placeholder blocking card or an + // agent-authored escaped-newline body. + const [senderRow] = await tx.select({ type: agents.type }).from(agents).where(eq(agents.uuid, senderId)).limit(1); + if (!senderRow) throw new NotFoundError(`Sender agent "${senderId}" not found`); + const effectiveContent = normalizeNonHumanTextContent({ + chatId, + senderId, + senderType: senderRow.type, + content: data.content, + }); + validateMessageContent( + { format: data.format ?? msg.format, content: effectiveContent }, + { hasAttachmentRefs: attachmentRefsFromMetadata(msg.metadata ?? undefined).length > 0 }, + ); + setClause.content = effectiveContent; + } + + // Patch only the edit timestamp in Postgres so concurrent server-owned + // metadata transitions cannot be overwritten by a stale read of the row. + setClause.metadata = sql`jsonb_set(${messages.metadata}, '{editedAt}', ${JSON.stringify( + new Date().toISOString(), + )}::jsonb)`; + + const [updatedRow] = await tx.update(messages).set(setClause).where(eq(messages.id, messageId)).returning(); + if (!updatedRow) throw new Error("Unexpected: UPDATE RETURNING produced no row"); + + // Reconcile the reference ledger against the post-edit shape. New + // content imageIds are validated + recorded; imageIds the edit dropped + // lose their edge, and an attachment left with zero edges is + // tombstoned here ("delete on last reference removed"). + const [chatRow] = await tx + .select({ organizationId: chats.organizationId }) + .from(chats) + .where(eq(chats.id, chatId)) + .limit(1); + if (!chatRow) throw new NotFoundError(`Chat "${chatId}" not found`); + const sync = await syncMessageAttachmentReferences(tx, { + messageId, + organizationId: chatRow.organizationId, + content: updatedRow.content, + metadata: updatedRow.metadata, }); - validateMessageContent( - { format: data.format ?? msg.format, content: effectiveContent }, - { hasAttachmentRefs: attachmentRefsFromMetadata(msg.metadata ?? undefined).length > 0 }, - ); - setClause.content = effectiveContent; - } - // Patch only the edit timestamp in Postgres so concurrent server-owned - // metadata transitions cannot be overwritten by a stale read of the row. - setClause.metadata = sql`jsonb_set(${messages.metadata}, '{editedAt}', ${JSON.stringify( - new Date().toISOString(), - )}::jsonb)`; + return { updated: updatedRow, removedForDeletion: sync.removedForDeletion }; + }); + + // Post-commit: destroy tombstoned attachments immediately (object first, + // then row). Best-effort — failures are logged and the background sweep + // retries; the tombstone already hides the attachment either way. + if (removedForDeletion.length > 0) { + await destroyDeletingAttachments(db, objectStorage, removedForDeletion); + } - const [updated] = await db.update(messages).set(setClause).where(eq(messages.id, messageId)).returning(); - if (!updated) throw new Error("Unexpected: UPDATE RETURNING produced no row"); return updated; } diff --git a/packages/server/src/services/object-storage.ts b/packages/server/src/services/object-storage.ts new file mode 100644 index 000000000..226bf2b98 --- /dev/null +++ b/packages/server/src/services/object-storage.ts @@ -0,0 +1,206 @@ +import type { Readable } from "node:stream"; +import { + CreateBucketCommand, + DeleteObjectCommand, + GetObjectCommand, + HeadBucketCommand, + PutObjectCommand, + S3Client, +} from "@aws-sdk/client-s3"; +import { getSignedUrl } from "@aws-sdk/s3-request-presigner"; +import { ATTACHMENT_PRESIGN_TTL_SECONDS } from "@first-tree/shared"; +import type { Config } from "../config.js"; +import { createLogger } from "../observability/logger.js"; + +const log = createLogger("ObjectStorage"); + +export type ObjectStorageConfig = NonNullable; + +/** Deterministic payload key for an attachment row. */ +export function attachmentObjectKey(attachmentId: string): string { + return `attachments/${attachmentId}`; +} + +/** Deterministic payload key for an agent avatar (overwritten in place). */ +export function avatarObjectKey(agentUuid: string): string { + return `avatars/${agentUuid}`; +} + +export type PutObjectStreamOptions = { + /** Exact payload length. Required — uploads reserve quota from the declared size. */ + contentLength: number; + contentType: string; + /** Aborts the in-flight PUT promptly (e.g. when the body stream failed). */ + abortSignal?: AbortSignal; +}; + +export type GetObjectStreamResult = { + body: Readable; + contentLength: number | undefined; +}; + +export type PresignGetOptions = { + /** Original filename, carried into `Content-Disposition` of the S3 response. */ + filename: string; + /** Logical MIME type, carried into `Content-Type` of the S3 response. */ + mimeType: string; + /** Disposition type; the download route serves inline like it always has. */ + disposition: "inline" | "attachment"; +}; + +/** + * Thin, S3-compatible object-storage boundary (AWS S3, Cloudflare R2, MinIO). + * + * Deliberately storage-dumb: keys are opaque strings owned by the callers + * (`attachmentObjectKey` / `avatarObjectKey`), lifecycle and quota decisions + * live in the attachment services, and every method maps 1:1 onto one S3 + * call so failure semantics stay predictable. + */ +export type ObjectStorage = { + /** + * Stream a payload into the bucket. The body is NOT buffered — callers + * pass the (possibly transformed) request stream plus the exact + * `contentLength`, and the SDK signs a single streaming PUT. + */ + putObjectStream(key: string, body: Readable, opts: PutObjectStreamOptions): Promise; + /** + * Open a payload stream. Returns `null` when the object does not exist — + * for a `stored` row that is corruption, and the caller decides how loud + * to be about it. + */ + getObjectStream(key: string): Promise; + /** Delete a payload. Idempotent: a missing object resolves silently. */ + deleteObject(key: string): Promise; + /** + * Presign a short-lived GET (redirect download mode). Signed against + * `publicEndpoint` when configured so browsers can reach storage across a + * split-horizon network; response content headers are pinned so the + * browser sees the original filename/MIME regardless of bucket metadata. + */ + presignGetUrl(key: string, opts: PresignGetOptions): Promise; + /** + * Best-effort bucket bootstrap for dev/test convenience: create the + * bucket when it is missing. Never throws — production deployments often + * scope credentials to object CRUD only and pre-provision the bucket, so + * a failed probe/create degrades to a warning and the first real + * operation surfaces the actual error. + */ + ensureBucket(): Promise; +}; + +function buildClient(config: ObjectStorageConfig, endpoint: string | undefined): S3Client { + return new S3Client({ + region: config.region, + ...(endpoint ? { endpoint } : {}), + forcePathStyle: config.forcePathStyle, + credentials: { + accessKeyId: config.accessKeyId, + secretAccessKey: config.secretAccessKey, + }, + // Skip the SDK's default CRC32 wrapping of every PutObject body: none of + // our calls require checksums, the extra internal body pipe leaks an + // unhandled rejection when an aborted streaming upload destroys the + // source mid-flight, and some S3-compatible backends reject the + // checksum trailers anyway. + requestChecksumCalculation: "WHEN_REQUIRED", + responseChecksumValidation: "WHEN_REQUIRED", + }); +} + +function isNotFoundError(error: unknown): boolean { + if (typeof error !== "object" || error === null) return false; + const name = "name" in error ? (error as { name?: unknown }).name : undefined; + if (name === "NoSuchKey" || name === "NotFound" || name === "NoSuchBucket") return true; + const status = + "$metadata" in error + ? (error as { $metadata?: { httpStatusCode?: unknown } }).$metadata?.httpStatusCode + : undefined; + return status === 404; +} + +export function createObjectStorage(config: ObjectStorageConfig): ObjectStorage { + const client = buildClient(config, config.endpoint); + // Separate client for presigning only: URLs must be reachable by browsers, + // which may live on the public side of a split-horizon network. + const presignClient = config.publicEndpoint ? buildClient(config, config.publicEndpoint) : client; + const bucket = config.bucket; + + return { + async putObjectStream(key, body, opts) { + await client.send( + new PutObjectCommand({ + Bucket: bucket, + Key: key, + Body: body, + ContentLength: opts.contentLength, + ContentType: opts.contentType, + }), + { abortSignal: opts.abortSignal }, + ); + }, + + async getObjectStream(key) { + try { + const response = await client.send(new GetObjectCommand({ Bucket: bucket, Key: key })); + const body = response.Body; + if (!body) return null; + // The SDK types Body as a browser/node union; under Node a GetObject + // body is always a Readable (SdkStream). + return { body: body as unknown as Readable, contentLength: response.ContentLength }; + } catch (error) { + if (isNotFoundError(error)) return null; + throw error; + } + }, + + async deleteObject(key) { + try { + await client.send(new DeleteObjectCommand({ Bucket: bucket, Key: key })); + } catch (error) { + // S3 DeleteObject already succeeds on missing keys; tolerate backends + // that surface 404/NoSuchKey instead so retries stay idempotent. + if (isNotFoundError(error)) return; + throw error; + } + }, + + async presignGetUrl(key, opts) { + const command = new GetObjectCommand({ + Bucket: bucket, + Key: key, + ResponseContentType: opts.mimeType, + ResponseContentDisposition: contentDisposition(opts.filename, opts.disposition), + }); + return getSignedUrl(presignClient, command, { expiresIn: ATTACHMENT_PRESIGN_TTL_SECONDS }); + }, + + async ensureBucket() { + try { + await client.send(new HeadBucketCommand({ Bucket: bucket })); + return; + } catch (error) { + if (!isNotFoundError(error)) { + log.warn({ err: error, bucket }, "object storage bucket probe failed; continuing"); + return; + } + } + try { + await client.send(new CreateBucketCommand({ Bucket: bucket })); + log.info({ bucket }, "created object storage bucket"); + } catch (error) { + log.warn({ err: error, bucket }, "object storage bucket create failed; continuing"); + } + }, + }; +} + +/** + * RFC 6266 / RFC 5987 Content-Disposition for arbitrary (possibly + * non-ASCII) filenames: an ASCII fallback plus a UTF-8 `filename*`. Used by + * the proxy download path and presigned URLs so both modes emit identical + * headers. + */ +export function contentDisposition(filename: string, disposition: "inline" | "attachment"): string { + const fallback = filename.replace(/[^\x20-\x7e]/g, "_").replace(/["\\]/g, "'"); + return `${disposition}; filename="${fallback}"; filename*=UTF-8''${encodeURIComponent(filename)}`; +} diff --git a/packages/server/src/services/stream-limit.ts b/packages/server/src/services/stream-limit.ts new file mode 100644 index 000000000..9d96e1231 --- /dev/null +++ b/packages/server/src/services/stream-limit.ts @@ -0,0 +1,85 @@ +import { Transform } from "node:stream"; + +export type StreamingUploadOptions = { + /** The byte-limit stream sitting between the source and the consumer. */ + limiter: Transform; + /** `pipeline(source, limiter)` — the producer half. */ + producer: Promise; + /** Starts the storage PUT consuming the limiter; must honor the signal. */ + startConsumer: (abortSignal: AbortSignal) => Promise; +}; + +/** + * Coordinate the two halves of a streaming upload so that EITHER failure + * settles BOTH promptly and exactly one error surfaces: + * + * - producer fails (byte-count mismatch, client abort) → the consumer's + * in-flight PUT is aborted via the signal instead of waiting out an SDK + * timeout on a half-fed body; + * - consumer fails (storage down) → the limiter is destroyed so the + * backpressured source→limiter pipeline can settle instead of stalling. + * + * Both halves are always awaited (a bare `Promise.all` would orphan the + * second rejection as an unhandledRejection), and the producer's error wins + * when both reject — it is the root cause; the consumer's is derived. + */ +export async function settleStreamingUpload(opts: StreamingUploadOptions): Promise { + const abort = new AbortController(); + const producer = opts.producer.catch((error: unknown) => { + abort.abort(); + throw error; + }); + const consumer = opts.startConsumer(abort.signal).catch((error: unknown) => { + opts.limiter.destroy(error instanceof Error ? error : new Error(String(error))); + throw error; + }); + const [consumerResult, producerResult] = await Promise.allSettled([consumer, producer]); + if (producerResult.status === "rejected") throw producerResult.reason; + if (consumerResult.status === "rejected") throw consumerResult.reason; +} + +export type ByteLimitStreamOptions = { + /** Exact byte count the stream must carry (the declared Content-Length). */ + expectedBytes: number; + /** + * Error to destroy the stream with when the source exceeds + * `expectedBytes`. A mismatch in either direction also fails the stream + * at EOF with the same factory — the declared length is a contract, not + * a hint (quota was reserved from it). + */ + makeMismatchError: (seenBytes: number) => Error; +}; + +/** + * Pass-through stream enforcing an exact byte count. Used between the + * request stream and the object-storage PUT so no payload can sneak past + * the size the quota reservation was made for: + * + * - more bytes than declared → destroys mid-flight (upload aborts); + * - fewer bytes than declared (truncated body / client abort) → fails at + * EOF, before the storage layer could treat a short object as complete. + * + * Node's HTTP parser already cuts request bodies at Content-Length, so the + * overshoot branch mostly guards non-HTTP callers and tests; the EOF check + * is the load-bearing half. + */ +export function createByteLimitStream(opts: ByteLimitStreamOptions): Transform { + let seenBytes = 0; + return new Transform({ + transform(chunk: Buffer, _encoding, callback) { + seenBytes += chunk.byteLength; + if (seenBytes > opts.expectedBytes) { + callback(opts.makeMismatchError(seenBytes)); + return; + } + callback(null, chunk); + }, + flush(callback) { + if (seenBytes !== opts.expectedBytes) { + callback(opts.makeMismatchError(seenBytes)); + return; + } + callback(); + }, + }); +} diff --git a/packages/server/src/services/upload-gate.ts b/packages/server/src/services/upload-gate.ts new file mode 100644 index 000000000..51486f2d7 --- /dev/null +++ b/packages/server/src/services/upload-gate.ts @@ -0,0 +1,47 @@ +import { ATTACHMENT_ERROR_CODES } from "@first-tree/shared"; +import { TooManyRequestsError } from "../errors.js"; + +export type UploadGate = { + /** + * Claim an upload slot for `uploaderId`. Throws `TooManyRequestsError` + * (429, `ATTACHMENT_CONCURRENCY_EXCEEDED`) when the uploader already + * holds `maxConcurrent` streams. The returned function releases the slot + * and MUST run exactly once (call it from a `finally`). + */ + acquire(uploaderId: string): () => void; +}; + +/** + * Per-uploader concurrency gate for streaming uploads. Bounds how many + * parallel upload streams one uploader identity may hold on THIS server + * instance — the uploader key is `humanAgentId`, so all of one person's + * agents share the budget, and multi-replica deployments multiply the + * bound by the replica count (same in-process scoping as + * @fastify/rate-limit's default store; PostgreSQL stays the only shared + * backend). The global request rate limiter still applies on top. + */ +export function createUploadGate(maxConcurrent: number): UploadGate { + const inFlight = new Map(); + return { + acquire(uploaderId) { + const current = inFlight.get(uploaderId) ?? 0; + if (current >= maxConcurrent) { + throw new TooManyRequestsError(`Too many concurrent attachment uploads (limit ${maxConcurrent} per uploader)`, { + code: ATTACHMENT_ERROR_CODES.concurrencyExceeded, + }); + } + inFlight.set(uploaderId, current + 1); + let released = false; + return () => { + if (released) return; + released = true; + const value = inFlight.get(uploaderId) ?? 0; + if (value <= 1) { + inFlight.delete(uploaderId); + } else { + inFlight.set(uploaderId, value - 1); + } + }; + }, + }; +} diff --git a/packages/server/src/types.ts b/packages/server/src/types.ts index c3d458491..3b1004a6b 100644 --- a/packages/server/src/types.ts +++ b/packages/server/src/types.ts @@ -2,6 +2,7 @@ import type { Database } from "./db/connection.js"; import type { UserScope } from "./scope/types.js"; import type { ConfigService } from "./services/config-service.js"; import type { Notifier } from "./services/notifier.js"; +import type { ObjectStorage } from "./services/object-storage.js"; import type { ResourcesService } from "./services/resources.js"; export type AgentIdentity = { @@ -16,6 +17,8 @@ declare module "fastify" { interface FastifyInstance { db: Database; config: import("./config.js").Config; + /** S3-compatible payload store; null until FIRST_TREE_S3_* is configured. */ + objectStorage: ObjectStorage | null; notifier: Notifier; configService: ConfigService; resourcesService: ResourcesService; diff --git a/packages/shared/src/config/server-config.ts b/packages/shared/src/config/server-config.ts index 411d44545..083ad7dd9 100644 --- a/packages/shared/src/config/server-config.ts +++ b/packages/shared/src/config/server-config.ts @@ -1,6 +1,7 @@ import { join } from "node:path"; import { z } from "zod"; import { logFormatSchema, logLevelSchema } from "../observability/logger-core.js"; +import { ORG_ATTACHMENT_QUOTA_BYTES, ORG_ATTACHMENT_QUOTA_COUNT } from "../schemas/attachment.js"; import { runtimeProviderSchema } from "../schemas/runtime-provider.js"; import { defaultDataDir } from "./resolver.js"; import { defineConfig, field, optional } from "./schema.js"; @@ -214,6 +215,90 @@ export const serverConfigSchema = defineConfig({ }), provider: field(z.enum(["docker", "external"]).default("docker")), }, + /** + * S3-compatible object storage for binary payloads (attachments, agent + * avatars). Optional group: setting any member env activates it, and the + * required members (bucket, credentials) then fail loudly when missing. + * When the group is absent the attachment upload surface degrades to 503 + * ("object storage not configured") while pre-migration bytea downloads + * keep working — see services/object-storage.ts for the boundary. + * + * Any S3-compatible backend works (AWS S3, Cloudflare R2, MinIO). MinIO + * needs `forcePathStyle: true`. + */ + objectStorage: optional({ + bucket: field(z.string().min(1), { env: "FIRST_TREE_S3_BUCKET" }), + accessKeyId: field(z.string().min(1), { env: "FIRST_TREE_S3_ACCESS_KEY_ID", secret: true }), + secretAccessKey: field(z.string().min(1), { env: "FIRST_TREE_S3_SECRET_ACCESS_KEY", secret: true }), + /** Custom endpoint URL (MinIO/R2). Omit for native AWS resolution. */ + endpoint: field(optionalTrimmedStringSchema, { env: "FIRST_TREE_S3_ENDPOINT" }), + region: field(z.string().min(1).default("us-east-1"), { env: "FIRST_TREE_S3_REGION" }), + forcePathStyle: field(z.boolean().default(false), { env: "FIRST_TREE_S3_FORCE_PATH_STYLE" }), + /** + * Endpoint used when presigning redirect-mode download URLs, for + * deployments where the server reaches storage over an internal address + * but browsers must use a public one. Falls back to `endpoint`. + */ + publicEndpoint: field(optionalTrimmedStringSchema, { env: "FIRST_TREE_S3_PUBLIC_ENDPOINT" }), + }), + /** + * Attachment governance knobs. Quota semantics are hard-reject (413/422, + * no soft-warn mode); the values are deploy-tunable with governed + * defaults. Zero is deliberately rejected for the quota fields: 0-as- + * unlimited would silently disable governance and 0-as-reject-all is an + * operational footgun — disablement is not a supported mode. + */ + attachments: { + /** + * How downloads serve S3-backed payloads. `proxy` streams bytes through + * the server and works with any bucket/network topology out of the box. + * `redirect` answers 302 with a short-lived presigned URL — cheaper at + * scale, but requires the bucket to be browser-reachable and to carry + * CORS config for the web app origin (attachments are fetched with + * authenticated XHR, not plain tags). + */ + downloadMode: field(z.enum(["proxy", "redirect"]).default("proxy"), { + env: "FIRST_TREE_ATTACHMENT_DOWNLOAD_MODE", + }), + orgQuotaBytes: field(z.coerce.number().int().positive().default(ORG_ATTACHMENT_QUOTA_BYTES), { + env: "FIRST_TREE_ATTACHMENT_ORG_QUOTA_BYTES", + }), + orgQuotaCount: field(z.coerce.number().int().positive().default(ORG_ATTACHMENT_QUOTA_COUNT), { + env: "FIRST_TREE_ATTACHMENT_ORG_QUOTA_COUNT", + }), + /** + * Orphan sweep cadence. 0 disables the background timer (tests drive the + * sweep explicitly). Runs concurrently on every replica — passes are + * SKIP LOCKED + idempotent, so no leader election is needed. + */ + sweepIntervalSeconds: field(z.coerce.number().int().nonnegative().default(900), { + env: "FIRST_TREE_ATTACHMENT_SWEEP_INTERVAL_SECONDS", + }), + /** + * How long an attachment may stay unreferenced after upload before the + * sweep deletes it (governed default: 24h). + */ + orphanGraceSeconds: field(z.coerce.number().int().positive().default(86_400), { + env: "FIRST_TREE_ATTACHMENT_ORPHAN_GRACE_SECONDS", + }), + /** + * How long a `pending` row (quota reservation whose upload never + * finalized) survives before the sweep reclaims it. Bounds reservation + * leakage from crashed uploads. + */ + pendingTtlSeconds: field(z.coerce.number().int().positive().default(3600), { + env: "FIRST_TREE_ATTACHMENT_PENDING_TTL_SECONDS", + }), + /** + * Max parallel upload streams per uploader identity (humanAgentId — all + * of one person's agents share the budget), per server instance. Guards + * a single client holding unbounded concurrent streams; the global + * rate limiter still applies on top. + */ + maxConcurrentUploadsPerUploader: field(z.coerce.number().int().positive().default(4), { + env: "FIRST_TREE_ATTACHMENT_MAX_CONCURRENT_UPLOADS_PER_UPLOADER", + }), + }, server: { port: field(z.number().default(8000), { env: "FIRST_TREE_PORT" }), host: field(z.string().default("127.0.0.1"), { env: "FIRST_TREE_HOST" }), diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index 57cf1c296..1b3bcd4b7 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -187,11 +187,15 @@ export { RUNTIME_STALE_MS, } from "./schemas/agent-status.js"; export { + ATTACHMENT_ERROR_CODES, ATTACHMENT_FILENAME_HEADER, ATTACHMENT_MIME_HEADER, + ATTACHMENT_PRESIGN_TTL_SECONDS, type AttachmentMetadata, attachmentMetadataSchema, MAX_ATTACHMENT_BYTES, + ORG_ATTACHMENT_QUOTA_BYTES, + ORG_ATTACHMENT_QUOTA_COUNT, type UploadAttachmentResponse, uploadAttachmentResponseSchema, } from "./schemas/attachment.js"; diff --git a/packages/shared/src/schemas/attachment.ts b/packages/shared/src/schemas/attachment.ts index 6eb887f5d..951846165 100644 --- a/packages/shared/src/schemas/attachment.ts +++ b/packages/shared/src/schemas/attachment.ts @@ -11,6 +11,40 @@ import { z } from "zod"; */ export const MAX_ATTACHMENT_BYTES = 10 * 1024 * 1024; +/** + * Default per-organization attachment quotas, enforced server-side as hard + * rejects (422 with `ATTACHMENT_QUOTA_EXCEEDED`) at upload time. The byte + * quota implements the governed "2 GB" limit as 2 GiB (2^31) to stay + * consistent with the binary units used by `MAX_ATTACHMENT_BYTES`. Both are + * deploy-time tunable via env; these constants are the defaults, and the + * reject-on-exceed semantics do not vary with the configured value. + */ +export const ORG_ATTACHMENT_QUOTA_BYTES = 2 * 1024 * 1024 * 1024; +export const ORG_ATTACHMENT_QUOTA_COUNT = 1000; + +/** + * Stable machine-readable error codes surfaced in the error response body + * (`{ error, code }`) for attachment governance rejections. Clients must + * match on `code`, not on the human-readable message. + */ +export const ATTACHMENT_ERROR_CODES = { + /** Single file exceeds `MAX_ATTACHMENT_BYTES` (HTTP 413). */ + tooLarge: "ATTACHMENT_TOO_LARGE", + /** Org byte or object-count quota exceeded (HTTP 422). */ + quotaExceeded: "ATTACHMENT_QUOTA_EXCEEDED", + /** Uploader holds too many parallel upload streams (HTTP 429). */ + concurrencyExceeded: "ATTACHMENT_CONCURRENCY_EXCEEDED", + /** Upload did not declare Content-Length (HTTP 411). */ + lengthRequired: "ATTACHMENT_LENGTH_REQUIRED", +} as const; + +/** + * Lifetime of presigned download URLs handed out in redirect mode. Fixed by + * the governance spec ("short-lived, at most 5 minutes"), deliberately not + * env-tunable. + */ +export const ATTACHMENT_PRESIGN_TTL_SECONDS = 300; + /** * Header name (case-insensitive) carrying the original filename on upload. * Octet-stream uploads do not carry a filename in `Content-Disposition`, so diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 6b1b02311..2b0956417 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -171,6 +171,12 @@ importers: '@autotelic/fastify-opentelemetry': specifier: ^0.23.0 version: 0.23.0(@opentelemetry/api@1.9.1) + '@aws-sdk/client-s3': + specifier: ^3.1093.0 + version: 3.1093.0 + '@aws-sdk/s3-request-presigner': + specifier: ^3.1093.0 + version: 3.1093.0 '@fastify/cors': specifier: ^11.2.0 version: 11.2.0 @@ -502,6 +508,82 @@ packages: peerDependencies: '@opentelemetry/api': ^1.0.0 + '@aws-sdk/checksums@3.1000.19': + resolution: {integrity: sha512-Hc4N100RdkuWshKBnhPzmpdftfi9mCLz+OHFELHM1QIgMH4QRUUWyWgfiebta/YX2Bd62wTcm3EqAP8TeXv0gA==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/client-s3@3.1093.0': + resolution: {integrity: sha512-7452vEdp/nihIBWijnmcTBujXEFfbs4F02wyBDGqmNr6pwyo5GmQorx0zQIVg8QGFLXiBvsWKXBhCdiBcxNnGA==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/core@3.976.0': + resolution: {integrity: sha512-0cjRaEdlVoOrsNb9pP5q1Syyc8pXw5xSj2Np2ryReRTr9FppIIRVSdZK4lbnfmc2Hvgux/xBOUU6baB7z8//uA==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/credential-provider-env@3.972.60': + resolution: {integrity: sha512-BAkxdoe7tpDDqCghGpuOeHQRbm/2znVvOQm0AvpQbA2tbfMN46doN4zx65fv85ImP3KADwc2zQPmbrlI9MPfMg==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/credential-provider-http@3.972.62': + resolution: {integrity: sha512-g/0fGqKTb9xpKdd9AtpmV5Eo3DFKbnkpA2+w0peISSlu7NfAoWOuYBFxsu+yWBtxU89ka55ezoZBCbFaS8pjYQ==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/credential-provider-ini@3.973.5': + resolution: {integrity: sha512-ylubazcRfq2TVus/qXucSXeC42Qdjp5HQxTu68K/BsdMiZlcSLD1zkpoCgApXZX1Y6YJhtGGs7ZHhO/GuIgBlw==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/credential-provider-login@3.972.67': + resolution: {integrity: sha512-CCygIKJ9YbI3n84OClSaSppkgKKHVj2TGT33c6FRORZrYNZQ1POmD+ip0FLYokiJAK7sSdc3YVkOsBm90oxWMQ==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/credential-provider-node@3.972.71': + resolution: {integrity: sha512-HIg7Q2osBzajQwL+1Vkyh2E7Gim3eTNb9RHIsOxDGjW0eZg4oEKtRs5sioCnc73ilhaOm4gX2lHVF8J7+nt2rg==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/credential-provider-process@3.972.60': + resolution: {integrity: sha512-YIo3f99hM43QdYG8hDzwGemnR/pU95b0kramqSJUTleCqaB7+HwKf7YZFHqvOgTqZTPx/mRmNIqoDRr3U0Z3Tw==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/credential-provider-sso@3.973.4': + resolution: {integrity: sha512-BPdmL8sSBOCv4ngZ+3LHxyc3CNqDCEK37CHioCk7zGrTMY5sUtkH8q+o6qA80nn6w3/fyBPGNE7OIRlmoOxRQA==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/credential-provider-web-identity@3.972.66': + resolution: {integrity: sha512-kSAziJboOmZmsR9/MTbiNjowl2BPes1bQuJpne4qAZ62ubi8fjfr/aupJSQje6udBoYxXTQbsL0e0kby2la3ng==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/middleware-sdk-s3@3.972.65': + resolution: {integrity: sha512-udwNhRfDTfCB98mAHjjgsnKQlxygB4e0X+Obne/XjJpvVsF0YCQC8ZErd/8Z6IPoLQjtiKHzwqEDbZiLrJEnOg==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/nested-clients@3.997.34': + resolution: {integrity: sha512-Y9REVrSwmLM+Qy6sZJ7ofMC2S3Hr3tPP/4CzL5U1olPP7OGoF+6+Px0E49cVQBtSxJtyeLJMf0UaBErfeSahAA==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/s3-request-presigner@3.1093.0': + resolution: {integrity: sha512-yEqbZRxq+ZkzrEr2oArnt9YZkdKW2SoZW4v4qpCAgOlxs6YJSRdJx3lOLaRwO3jsXfuVgohvm6Bt9MD4qATilg==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/signature-v4-multi-region@3.996.41': + resolution: {integrity: sha512-QMUytg+FQMGouc8gHS00KoYih3+N6cqmVI/pQGOIo7Nr7OpQaiXjSYOuL+vsPZ1tymY4LAQ8MYcHJmws5LRxng==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/token-providers@3.1092.0': + resolution: {integrity: sha512-hBYUAr6iBLNFcsiWTgtBb0stdSw39VOUq4Sp4A5caCNf66BAZplWN4FleKrVpJx5li2YgdnK2DqoFSMWC642FQ==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/types@3.974.2': + resolution: {integrity: sha512-3W6IUtSxFbH6X7Wb7DzGCV5QiFQsd0g8bOfntpmDxQlzBoKWUMBu/JPQR0DwkE+Hpnxd6db1tXbOwdeHddG6cA==} + engines: {node: '>=20.0.0'} + + '@aws-sdk/xml-builder@3.972.36': + resolution: {integrity: sha512-RdGmS1GLrtaTOLE1ElSluMldNrpk9Emq6uYs8SS8iHlu5xTAmM9rRkM91o48+rIRryBtyO9t+uLYCoMG6jVMVA==} + engines: {node: '>=20.0.0'} + + '@aws/lambda-invoke-store@0.3.0': + resolution: {integrity: sha512-sl4Bm6yiMNYrZKkqqDFWN0UfnWhlS8ivKxrYl+6t0gCLrqr8y3B2IqZZbFRkfaVVp7C/baApyh71P+LeE1A2sQ==} + engines: {node: '>=18.0.0'} + '@babel/code-frame@7.29.0': resolution: {integrity: sha512-9NhCeYjq9+3uxgdtp20LSiJXJvN0FeCtNGpJxuMFZ1Kv3cWUNb6DOhJwUvcVCzKGR66cw4njwM6hrJLqgOwbcw==} engines: {node: '>=6.9.0'} @@ -687,11 +769,11 @@ packages: '@esbuild-kit/core-utils@3.3.2': resolution: {integrity: sha512-sPRAnw9CdSsRmEtnsl2WXWdyquogVpB3yZ3dgwJfe8zrOzTsV7cJvmwrKVa+0ma5BoiGJ+BoqkMvawbayKUsqQ==} - deprecated: 'Merged into tsx: https://tsx.hirok.io' + deprecated: 'Merged into tsx: https://tsx.is' '@esbuild-kit/esm-loader@2.6.5': resolution: {integrity: sha512-FxEMIkJKnodyA1OaCUoEvbYRkoZlLZ4d/eXFu9Fh8CbBBgP5EmZxrfTRyN0qpXZ4vOvqnE5YdRdcrmUUXuU+dA==} - deprecated: 'Merged into tsx: https://tsx.hirok.io' + deprecated: 'Merged into tsx: https://tsx.is' '@esbuild/aix-ppc64@0.25.12': resolution: {integrity: sha512-Hhmwd6CInZ3dwpuGTF8fJG6yoWmsToE+vYgD4nytZVxcu1ulHpUQRAB1UJ8+N1Am3Mz4+xOByoQoSZf4D+CpkA==} @@ -2546,6 +2628,30 @@ packages: resolution: {integrity: sha512-qcoSzo4n2MulVQ70UUPLq6dTleb2a2HwL2wuwvAgWhPChrYTuk6A6mDg6aQb9fairPAwFPiU9PzOANpoDJcz1A==} engines: {node: '>= 18'} + '@smithy/core@3.29.7': + resolution: {integrity: sha512-BiEE2bnnGoPKdlGe3L+gOYORDHFGPuYVRLP7iUow/Sflm0B4hC4XY3FC1MRuc7ltzpW2xNnXopKi34TTkULlKQ==} + engines: {node: '>=18.0.0'} + + '@smithy/credential-provider-imds@4.4.12': + resolution: {integrity: sha512-ZZPDbl/aRp77aycuoMlo3BTayT4CE2a3uoqETYZU5ySnVbhpl5IJiY7dCZedn+ZusyDLqVv44IvKBiXd2/nK0Q==} + engines: {node: '>=18.0.0'} + + '@smithy/fetch-http-handler@5.6.9': + resolution: {integrity: sha512-EJktha5m5MXCwzdXrlWyqb9UCNHNFKlg+PmTpRsdX3dncJPTiqYleM9OKj2mLgdVJHR01d2tU4alG+z2NdH5rQ==} + engines: {node: '>=18.0.0'} + + '@smithy/node-http-handler@4.9.9': + resolution: {integrity: sha512-xVBZ3hptB99iNO9XyWqEhC7KD9bP9UPXhuy3h5Y2ItCfBv160D9IIC/Fmmp3EbnWwit4C+KVqlSE+E29Nk/pPg==} + engines: {node: '>=18.0.0'} + + '@smithy/signature-v4@5.6.8': + resolution: {integrity: sha512-iGBm6hIwD2MGvVRSgrjVWa4FXtXDq3akxu0DCpnkmBo0xtEHZ/siMRt7ycfZAefYr2UdywUgmGtoRLaq5u56pg==} + engines: {node: '>=18.0.0'} + + '@smithy/types@4.16.1': + resolution: {integrity: sha512-0JFs3V2y2M9tKW5na/qxe69Zv+uxLMO7QBbhxF/FHu/Gp2NFZAAL9tWl9PU02xxo07pb3G9FTyjNc6D5uZrJIg==} + engines: {node: '>=18.0.0'} + '@stablelib/base64@1.0.1': resolution: {integrity: sha512-1bnPQqSxSuc3Ii6MhBysoWCg58j97aUjuCSZrGSmDxNqtytIi0k8utUenAwTZN4V5mXXYGsVUI9zeBqy+jBOSQ==} @@ -3057,6 +3163,9 @@ packages: resolution: {integrity: sha512-2cGmJupaNgg+QUwVLAucDuWuoMZ6EX9iHDRswZ5lsNYEmwPaRknMPCLZz07yTzVq/83p4o/wzbDZbBrTvGGTIw==} engines: {node: '>=18'} + bowser@2.14.1: + resolution: {integrity: sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg==} + brace-expansion@2.0.2: resolution: {integrity: sha512-Jt0vHyM+jmUBqojB7E1NIYadt0vI0Qxjxd2TErW94wDz+E2LAm5vKMXXwg6ZZBTHPuUlDgQHKXvjGBdfcF1ZDQ==} @@ -5364,6 +5473,180 @@ snapshots: '@opentelemetry/api': 1.9.1 fastify-plugin: 5.1.0 + '@aws-sdk/checksums@3.1000.19': + dependencies: + '@aws-sdk/core': 3.976.0 + '@aws-sdk/types': 3.974.2 + '@smithy/core': 3.29.7 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/client-s3@3.1093.0': + dependencies: + '@aws-sdk/checksums': 3.1000.19 + '@aws-sdk/core': 3.976.0 + '@aws-sdk/credential-provider-node': 3.972.71 + '@aws-sdk/middleware-sdk-s3': 3.972.65 + '@aws-sdk/signature-v4-multi-region': 3.996.41 + '@aws-sdk/types': 3.974.2 + '@smithy/core': 3.29.7 + '@smithy/fetch-http-handler': 5.6.9 + '@smithy/node-http-handler': 4.9.9 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/core@3.976.0': + dependencies: + '@aws-sdk/types': 3.974.2 + '@aws-sdk/xml-builder': 3.972.36 + '@aws/lambda-invoke-store': 0.3.0 + '@smithy/core': 3.29.7 + '@smithy/signature-v4': 5.6.8 + '@smithy/types': 4.16.1 + bowser: 2.14.1 + tslib: 2.8.1 + + '@aws-sdk/credential-provider-env@3.972.60': + dependencies: + '@aws-sdk/core': 3.976.0 + '@aws-sdk/types': 3.974.2 + '@smithy/core': 3.29.7 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/credential-provider-http@3.972.62': + dependencies: + '@aws-sdk/core': 3.976.0 + '@aws-sdk/types': 3.974.2 + '@smithy/core': 3.29.7 + '@smithy/fetch-http-handler': 5.6.9 + '@smithy/node-http-handler': 4.9.9 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/credential-provider-ini@3.973.5': + dependencies: + '@aws-sdk/core': 3.976.0 + '@aws-sdk/credential-provider-env': 3.972.60 + '@aws-sdk/credential-provider-http': 3.972.62 + '@aws-sdk/credential-provider-login': 3.972.67 + '@aws-sdk/credential-provider-process': 3.972.60 + '@aws-sdk/credential-provider-sso': 3.973.4 + '@aws-sdk/credential-provider-web-identity': 3.972.66 + '@aws-sdk/nested-clients': 3.997.34 + '@aws-sdk/types': 3.974.2 + '@smithy/core': 3.29.7 + '@smithy/credential-provider-imds': 4.4.12 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/credential-provider-login@3.972.67': + dependencies: + '@aws-sdk/core': 3.976.0 + '@aws-sdk/nested-clients': 3.997.34 + '@aws-sdk/types': 3.974.2 + '@smithy/core': 3.29.7 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/credential-provider-node@3.972.71': + dependencies: + '@aws-sdk/credential-provider-env': 3.972.60 + '@aws-sdk/credential-provider-http': 3.972.62 + '@aws-sdk/credential-provider-ini': 3.973.5 + '@aws-sdk/credential-provider-process': 3.972.60 + '@aws-sdk/credential-provider-sso': 3.973.4 + '@aws-sdk/credential-provider-web-identity': 3.972.66 + '@aws-sdk/types': 3.974.2 + '@smithy/core': 3.29.7 + '@smithy/credential-provider-imds': 4.4.12 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/credential-provider-process@3.972.60': + dependencies: + '@aws-sdk/core': 3.976.0 + '@aws-sdk/types': 3.974.2 + '@smithy/core': 3.29.7 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/credential-provider-sso@3.973.4': + dependencies: + '@aws-sdk/core': 3.976.0 + '@aws-sdk/nested-clients': 3.997.34 + '@aws-sdk/token-providers': 3.1092.0 + '@aws-sdk/types': 3.974.2 + '@smithy/core': 3.29.7 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/credential-provider-web-identity@3.972.66': + dependencies: + '@aws-sdk/core': 3.976.0 + '@aws-sdk/nested-clients': 3.997.34 + '@aws-sdk/types': 3.974.2 + '@smithy/core': 3.29.7 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/middleware-sdk-s3@3.972.65': + dependencies: + '@aws-sdk/core': 3.976.0 + '@aws-sdk/signature-v4-multi-region': 3.996.41 + '@aws-sdk/types': 3.974.2 + '@smithy/core': 3.29.7 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/nested-clients@3.997.34': + dependencies: + '@aws-sdk/core': 3.976.0 + '@aws-sdk/signature-v4-multi-region': 3.996.41 + '@aws-sdk/types': 3.974.2 + '@smithy/core': 3.29.7 + '@smithy/fetch-http-handler': 5.6.9 + '@smithy/node-http-handler': 4.9.9 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/s3-request-presigner@3.1093.0': + dependencies: + '@aws-sdk/core': 3.976.0 + '@aws-sdk/signature-v4-multi-region': 3.996.41 + '@aws-sdk/types': 3.974.2 + '@smithy/core': 3.29.7 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/signature-v4-multi-region@3.996.41': + dependencies: + '@aws-sdk/types': 3.974.2 + '@smithy/signature-v4': 5.6.8 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/token-providers@3.1092.0': + dependencies: + '@aws-sdk/core': 3.976.0 + '@aws-sdk/nested-clients': 3.997.34 + '@aws-sdk/types': 3.974.2 + '@smithy/core': 3.29.7 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/types@3.974.2': + dependencies: + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws-sdk/xml-builder@3.972.36': + dependencies: + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@aws/lambda-invoke-store@0.3.0': {} + '@babel/code-frame@7.29.0': dependencies: '@babel/helper-validator-identifier': 7.28.5 @@ -7326,6 +7609,39 @@ snapshots: - rollup - supports-color + '@smithy/core@3.29.7': + dependencies: + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@smithy/credential-provider-imds@4.4.12': + dependencies: + '@smithy/core': 3.29.7 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@smithy/fetch-http-handler@5.6.9': + dependencies: + '@smithy/core': 3.29.7 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@smithy/node-http-handler@4.9.9': + dependencies: + '@smithy/core': 3.29.7 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@smithy/signature-v4@5.6.8': + dependencies: + '@smithy/core': 3.29.7 + '@smithy/types': 4.16.1 + tslib: 2.8.1 + + '@smithy/types@4.16.1': + dependencies: + tslib: 2.8.1 + '@stablelib/base64@1.0.1': {} '@tailwindcss/node@4.2.2': @@ -7674,6 +7990,22 @@ snapshots: chai: 5.3.3 tinyrainbow: 2.0.0 + '@vitest/mocker@3.2.4(vite@6.4.1(@types/node@22.19.15)(jiti@2.6.1)(lightningcss@1.32.0)(tsx@4.21.0)(yaml@2.8.2))': + dependencies: + '@vitest/spy': 3.2.4 + estree-walker: 3.0.3 + magic-string: 0.30.21 + optionalDependencies: + vite: 6.4.1(@types/node@22.19.15)(jiti@2.6.1)(lightningcss@1.32.0)(tsx@4.21.0)(yaml@2.8.2) + + '@vitest/mocker@3.2.4(vite@6.4.1(@types/node@22.19.15)(jiti@2.6.1)(lightningcss@1.32.0)(tsx@4.21.0)(yaml@2.8.3))': + dependencies: + '@vitest/spy': 3.2.4 + estree-walker: 3.0.3 + magic-string: 0.30.21 + optionalDependencies: + vite: 6.4.1(@types/node@22.19.15)(jiti@2.6.1)(lightningcss@1.32.0)(tsx@4.21.0)(yaml@2.8.3) + '@vitest/mocker@3.2.4(vite@6.4.1(@types/node@22.19.19)(jiti@2.6.1)(lightningcss@1.32.0)(tsx@4.21.0)(yaml@2.8.3))': dependencies: '@vitest/spy': 3.2.4 @@ -7899,6 +8231,8 @@ snapshots: transitivePeerDependencies: - supports-color + bowser@2.14.1: {} + brace-expansion@2.0.2: dependencies: balanced-match: 1.0.2 @@ -10342,7 +10676,7 @@ snapshots: dependencies: '@types/chai': 5.2.3 '@vitest/expect': 3.2.4 - '@vitest/mocker': 3.2.4(vite@6.4.1(@types/node@22.19.19)(jiti@2.6.1)(lightningcss@1.32.0)(tsx@4.21.0)(yaml@2.8.3)) + '@vitest/mocker': 3.2.4(vite@6.4.1(@types/node@22.19.15)(jiti@2.6.1)(lightningcss@1.32.0)(tsx@4.21.0)(yaml@2.8.2)) '@vitest/pretty-format': 3.2.4 '@vitest/runner': 3.2.4 '@vitest/snapshot': 3.2.4 @@ -10385,7 +10719,7 @@ snapshots: dependencies: '@types/chai': 5.2.3 '@vitest/expect': 3.2.4 - '@vitest/mocker': 3.2.4(vite@6.4.1(@types/node@22.19.19)(jiti@2.6.1)(lightningcss@1.32.0)(tsx@4.21.0)(yaml@2.8.3)) + '@vitest/mocker': 3.2.4(vite@6.4.1(@types/node@22.19.15)(jiti@2.6.1)(lightningcss@1.32.0)(tsx@4.21.0)(yaml@2.8.3)) '@vitest/pretty-format': 3.2.4 '@vitest/runner': 3.2.4 '@vitest/snapshot': 3.2.4