diff --git a/.mintignore b/.mintignore index d2c59d971..75266d2b2 100644 --- a/.mintignore +++ b/.mintignore @@ -10,3 +10,19 @@ scripts/ # build (e.g. "Could not parse import/exports with acorn"). .pipeline-scratch/ .claude/ + +# Internal repo docs, authoring templates, and the repo README — not docs pages. +# These were being published as /CLAUDE, /README, /SKILLS_SYNC_PIPELINE, and +# /WIREFRAME_VARIABLES_TEMPLATE on the live site. +/CLAUDE.md +/README.md +/SKILLS_SYNC_PIPELINE.md +/WIREFRAME_VARIABLES_TEMPLATE.md + +# Scratch release-notes pipeline artifacts left in the repo root. +/temp-release-notes.mdx +/temp-release-notes-input.mdx + +# GitHub workflow + automation docs (e.g. the release-note pipeline README, +# which references internal repositories). +.github/ diff --git a/SKILLS_SYNC_PIPELINE.md b/SKILLS_SYNC_PIPELINE.md index 545abd810..f9e4e4703 100644 --- a/SKILLS_SYNC_PIPELINE.md +++ b/SKILLS_SYNC_PIPELINE.md @@ -12,7 +12,7 @@ Every push to `velt-js/docs/main` touching one of the watched paths runs `.githu ``` async-collaboration/** realtime-collaboration/** -api-reference/rest-apis/** self-host-data/** +api-reference/rest-apis/** self-hosting/partial/** security/** webhooks/** get-started/** backend-sdks/** ui-customization/** permission-management/** diff --git a/api-reference/rest-apis/v2/notifications/add-notifications.mdx b/api-reference/rest-apis/v2/notifications/add-notifications.mdx index 2613c02ea..94c43c4a2 100644 --- a/api-reference/rest-apis/v2/notifications/add-notifications.mdx +++ b/api-reference/rest-apis/v2/notifications/add-notifications.mdx @@ -59,7 +59,7 @@ Use this API to add notifications. - Set to `true` to enable [Notification Resolver](/self-host-data/notifications) mode. + Set to `true` to enable [Notification Resolver](/self-hosting/partial/notifications) mode. When enabled, `displayHeadlineMessageTemplate` and `displayBodyMessage` are not required — notification content is resolved from your own infrastructure at read time. @@ -99,7 +99,7 @@ Use this API to add notifications. - Source identifier for the notification. Set to `'custom'` for notifications that should be routed through the [Notification Resolver](/self-host-data/notifications). Other values include `'comment'`, `'huddle'`, and `'crdt'`. + Source identifier for the notification. Set to `'custom'` for notifications that should be routed through the [Notification Resolver](/self-hosting/partial/notifications). Other values include `'comment'`, `'huddle'`, and `'crdt'`. @@ -247,7 +247,7 @@ When context is provided, notifications will be filtered based on the specified ## **Example Request with Notification Resolver** -Use this when notification content (headline, body) is stored on your own infrastructure and resolved at read time via the [Notification Resolver](/self-host-data/notifications). Omit `displayHeadlineMessageTemplate` and `displayBodyMessage` and set `notificationSource` to `'custom'`. +Use this when notification content (headline, body) is stored on your own infrastructure and resolved at read time via the [Notification Resolver](/self-hosting/partial/notifications). Omit `displayHeadlineMessageTemplate` and `displayBodyMessage` and set `notificationSource` to `'custom'`. ```JSON { diff --git a/api-reference/sdk/api/api-methods.mdx b/api-reference/sdk/api/api-methods.mdx index e935c7f91..2866adbbb 100644 --- a/api-reference/sdk/api/api-methods.mdx +++ b/api-reference/sdk/api/api-methods.mdx @@ -3839,7 +3839,7 @@ Set the data providers for self hosting data. - Params: [`VeltDataProvider`](/api-reference/sdk/models/data-models#veltdataprovider) - Returns: `void` - React Hook: `n/a` -- [Full Documentation →](/self-host-data/overview) +- [Full Documentation →](/self-hosting/partial/overview) #### setAnonymousUserDataProvider() Register a provider to resolve email → userId mappings for anonymous users tagged by email in comments. diff --git a/api-reference/sdk/models/data-models.mdx b/api-reference/sdk/models/data-models.mdx index 24074ae4d..002c46c43 100644 --- a/api-reference/sdk/models/data-models.mdx +++ b/api-reference/sdk/models/data-models.mdx @@ -4524,7 +4524,7 @@ Self-hosting configuration that maps Velt collections to custom Firestore paths. | Property | Type | Required | Description | | ----------------------- | -------- | -------- | ---------------------------------------------------------------------------------------------------------- | -| `userNotificationIndex` | `string` | No | Path to the global, per-user notification index collection (org-independent). Required for cross-organization "For You" notifications. See [Cross-Organization Notifications](/self-host-data/notifications#cross-organization-notifications). | +| `userNotificationIndex` | `string` | No | Path to the global, per-user notification index collection (org-independent). Required for cross-organization "For You" notifications. See [Cross-Organization Notifications](/self-hosting/partial/notifications#cross-organization-notifications). | #### NotificationSettingsLayout @@ -5809,7 +5809,7 @@ Element type of [`ResolverConfig.additionalSaveEvents`](#resolverconfig). Enum defining module names for user-related resolver operations. Used in data provider events to identify which module triggered the resolver call. -[Self-Host User Data →](/self-host-data/users#debugging) +[Self-Host User Data →](/self-hosting/partial/users#debugging) | Name | Value | Description | | ------------------------- | ------------------------- | -------------------------------------------------------- | @@ -5825,7 +5825,7 @@ Enum defining module names for user-related resolver operations. Used in data pr Enum defining module names for comment-related resolver operations. Used in data provider events to identify which module triggered the resolver call. -[Self-Host Comment Data →](/self-host-data/comments#debugging) +[Self-Host Comment Data →](/self-hosting/partial/comments#debugging) | Name | Value | Description | | ------------------------- | ------------------------- | -------------------------------------------------- | @@ -5839,7 +5839,7 @@ Enum defining module names for comment-related resolver operations. Used in data Enum defining module names for reaction-related resolver operations. Used in data provider events to identify which module triggered the resolver call. -[Self-Host Reaction Data →](/self-host-data/reactions#debugging) +[Self-Host Reaction Data →](/self-hosting/partial/reactions#debugging) | Name | Value | Description | | -------------------------- | -------------------------- | ------------------------------------------------- | @@ -5852,7 +5852,7 @@ Enum defining module names for reaction-related resolver operations. Used in dat Enum defining module names for recorder-related resolver operations. Used in data provider events to identify which module triggered the resolver call. -[Self-Host Recording Data →](/self-host-data/recordings#debugging) +[Self-Host Recording Data →](/self-hosting/partial/recordings#debugging) | Name | Value | Description | | -------------------------- | -------------------------- | ------------------------------------------------- | diff --git a/async-collaboration/comments/customize-behavior.mdx b/async-collaboration/comments/customize-behavior.mdx index 0e5eb4504..68db9652f 100644 --- a/async-collaboration/comments/customize-behavior.mdx +++ b/async-collaboration/comments/customize-behavior.mdx @@ -2240,7 +2240,7 @@ contactElement.updateContactListScopeForOrganizationUsers(['all', 'organization' Register a provider to resolve email → `userId` mappings for users who are tagged by email in comments but are not part of the contact list. When a comment is saved and a tagged contact or `to` recipient has an email but no `userId`, the SDK calls the provider to look up the `userId` and backfills it into the comment data before persisting. -[Full implementation guide →](/self-host-data/users#anonymous-user-resolution) +[Full implementation guide →](/self-hosting/partial/users#anonymous-user-resolution) #### autoCompleteScrollConfig diff --git a/backend-sdks/python.mdx b/backend-sdks/python.mdx index 5a35f7ccb..7c333cc68 100644 --- a/backend-sdks/python.mdx +++ b/backend-sdks/python.mdx @@ -5226,4 +5226,4 @@ sdk.selfHosting.comments.saveComments(request) ## Resources - [Velt-Py PyPI Package](https://pypi.org/project/velt-py) -- [Self-Host Data Documentation](/self-host-data/overview) +- [Partial Self Hosting Documentation](/self-hosting/partial/overview) diff --git a/docs.json b/docs.json index 26cbed0d5..90dbc4755 100644 --- a/docs.json +++ b/docs.json @@ -11,7 +11,14 @@ "ignore": [ ".claude/**", "node_modules/**", - "scripts/**" + "scripts/**", + ".github/**", + "CLAUDE.md", + "README.md", + "SKILLS_SYNC_PIPELINE.md", + "WIREFRAME_VARIABLES_TEMPLATE.md", + "temp-release-notes.mdx", + "temp-release-notes-input.mdx" ], "navigation": { "tabs": [ @@ -341,17 +348,39 @@ ] }, { - "group": "Self-Host Data", + "group": "Self Hosting", "pages": [ - "self-host-data/overview", - "self-host-data/users", - "self-host-data/comments", - "self-host-data/reactions", - "self-host-data/attachments", - "self-host-data/recordings", - "self-host-data/activity", - "self-host-data/notifications", - "self-host-data/field-inventory" + { + "group": "Partial Self Hosting", + "pages": [ + "self-hosting/partial/overview", + "self-hosting/partial/users", + "self-hosting/partial/comments", + "self-hosting/partial/reactions", + "self-hosting/partial/attachments", + "self-hosting/partial/recordings", + "self-hosting/partial/activity", + "self-hosting/partial/notifications", + "self-hosting/partial/field-inventory" + ] + }, + { + "group": "Full Self Hosting", + "pages": [ + "self-hosting/full/overview", + { + "group": "GCP", + "pages": [ + "self-hosting/full/gcp/overview", + "self-hosting/full/gcp/install", + "self-hosting/full/gcp/upgrade", + "self-hosting/full/gcp/reference" + ] + }, + "self-hosting/full/aws", + "self-hosting/full/azure" + ] + } ] }, { @@ -1632,6 +1661,58 @@ { "source": "/realtime-collaboration/crdt/version-history", "destination": "/realtime-collaboration/crdt/overview" + }, + { + "source": "/self-host-data/overview", + "destination": "/self-hosting/partial/overview" + }, + { + "source": "/self-host-data/users", + "destination": "/self-hosting/partial/users" + }, + { + "source": "/self-host-data/comments", + "destination": "/self-hosting/partial/comments" + }, + { + "source": "/self-host-data/reactions", + "destination": "/self-hosting/partial/reactions" + }, + { + "source": "/self-host-data/attachments", + "destination": "/self-hosting/partial/attachments" + }, + { + "source": "/self-host-data/recordings", + "destination": "/self-hosting/partial/recordings" + }, + { + "source": "/self-host-data/activity", + "destination": "/self-hosting/partial/activity" + }, + { + "source": "/self-host-data/notifications", + "destination": "/self-hosting/partial/notifications" + }, + { + "source": "/self-host-data/field-inventory", + "destination": "/self-hosting/partial/field-inventory" + }, + { + "source": "/self-host-data", + "destination": "/self-hosting/partial/overview" + }, + { + "source": "/self-hosting", + "destination": "/self-hosting/partial/overview" + }, + { + "source": "/self-hosting/full", + "destination": "/self-hosting/full/overview" + }, + { + "source": "/self-hosting/full/gcp", + "destination": "/self-hosting/full/gcp/overview" } ], "logo": { diff --git a/release-notes/version-4/sdk-changelog.mdx b/release-notes/version-4/sdk-changelog.mdx index 4410f02a4..a42214c6c 100644 --- a/release-notes/version-4/sdk-changelog.mdx +++ b/release-notes/version-4/sdk-changelog.mdx @@ -871,7 +871,7 @@ commentElement.on('composerTextChange').subscribe((event) => { ### New Features -- [**Self-hosting**]: Added attachment resolver support for config-based endpoints. Configure `saveConfig` and `deleteConfig` URLs for file uploads and deletions. [Learn more](/self-host-data/attachments) +- [**Self-hosting**]: Added attachment resolver support for config-based endpoints. Configure `saveConfig` and `deleteConfig` URLs for file uploads and deletions. [Learn more](/self-hosting/partial/attachments) @@ -5733,7 +5733,7 @@ Note after this update, if you plan to go back to older version of the SDK befor ### New Features - [**Self-hosting**]: Added support to disable getting user resolver requests for organization, document and folder users. Helps optimize performance by avoiding unnecessary user data requests - Sometimes you may have a lot of users within your organization, document or folder. In such cases, you may want to disable user resolver requests for organization, document and folder users and instead use [custom autocomplete feature](/async-collaboration/comments/customize-behavior#customautocompletesearch). - - Learn more about [user data provider](/self-host-data/users) + - Learn more about [user data provider](/self-hosting/partial/users) diff --git a/release-notes/version-5/sdk-changelog.mdx b/release-notes/version-5/sdk-changelog.mdx index 8efcfd941..87cc04c27 100644 --- a/release-notes/version-5/sdk-changelog.mdx +++ b/release-notes/version-5/sdk-changelog.mdx @@ -13,11 +13,11 @@ description: Release Notes of changes added to the core Velt SDK ### New Features -- [**Self-Hosting**]: Data provider headers now accepts a dynamic header that runs on every request and retry, so you can pass short-lived per-request tokens. A new optional `credentials` field enables cookie/session auth on cross-origin endpoints. Works with all endpoint-based resolvers and is backward-compatible. [Learn more →](/self-host-data/overview#async-headers-and-credentials) +- [**Self-Hosting**]: Data provider headers now accepts a dynamic header that runs on every request and retry, so you can pass short-lived per-request tokens. A new optional `credentials` field enables cookie/session auth on cross-origin endpoints. Works with all endpoint-based resolvers and is backward-compatible. [Learn more →](/self-hosting/partial/overview#async-headers-and-credentials) -- [**Self-Hosting**]: New opt-in `additionalSaveEvents` field on `ResolverConfig` sends extra annotation-level events (status change, priority change, assign, approve, reactions, subscribe, and more) to your existing save endpoint. The new `CommentResolverSaveEvent` enum lists all 14 supported events. Opt-in only — no extra events are sent unless you enable them. [Learn more →](/self-host-data/comments) +- [**Self-Hosting**]: New opt-in `additionalSaveEvents` field on `ResolverConfig` sends extra annotation-level events (status change, priority change, assign, approve, reactions, subscribe, and more) to your existing save endpoint. The new `CommentResolverSaveEvent` enum lists all 14 supported events. Opt-in only — no extra events are sent unless you enable them. [Learn more →](/self-hosting/partial/comments) -- [**Self-Hosting**]: `SaveCommentResolverRequest` now includes an optional `targetComment` with the full `PartialComment` when a save targets a specific comment, so your backend doesn't have to look it up from `commentId`. [Learn more →](/self-host-data/comments) +- [**Self-Hosting**]: `SaveCommentResolverRequest` now includes an optional `targetComment` with the full `PartialComment` when a save targets a specific comment, so your backend doesn't have to look it up from `commentId`. [Learn more →](/self-hosting/partial/comments) ### Improvements @@ -865,7 +865,7 @@ import { VeltActivityLog } from '@veltdev/react'; - [**Activity Logs**]: `targetEntityId` is now optional for non-`custom` `featureType` values (`comment`, `reaction`, `recorder`, `crdt`) when creating activities. An optional `id` field is also supported to use as the DB record ID for idempotent writes. `targetEntityId` remains required when `featureType` is `custom`. [Learn more →](/async-collaboration/activity/overview) - [**Access Control**]: [`getUserPermissions`](/api-reference/sdk/api/api-methods#getuserpermissions) response entries for folders and documents now include an `accessType` field (`public`, `restricted`, `organizationPrivate`). Documents in folders inherit their folder's `accessType`. The field is also present on `PERMISSION_DENIED` entries so consumers can inspect the effective access type even when access is denied. Organization-level access is now verified via DB permission rules rather than array membership checks. -- [**Self-Host Data**]: Added `activity` field to [`VeltDataProvider`](/api-reference/sdk/models/data-models#veltdataprovider) accepting an [`ActivityAnnotationDataProvider`](/api-reference/sdk/models/data-models#activityannotationdataprovider). When configured via [`setDataProviders()`](/api-reference/sdk/api/api-methods#setdataproviders), Velt strips PII from activity records before writing to its backend and routes it to your storage; on read, resolved data is merged back transparently. [Learn more →](/self-host-data/activity) +- [**Self-Host Data**]: Added `activity` field to [`VeltDataProvider`](/api-reference/sdk/models/data-models#veltdataprovider) accepting an [`ActivityAnnotationDataProvider`](/api-reference/sdk/models/data-models#activityannotationdataprovider). When configured via [`setDataProviders()`](/api-reference/sdk/api/api-methods#setdataproviders), Velt strips PII from activity records before writing to its backend and routes it to your storage; on read, resolved data is merged back transparently. [Learn more →](/self-hosting/partial/activity) @@ -965,7 +965,7 @@ Velt.setDataProviders({ -- [**Notifications**]: Added self-hosting options for custom notifications. When configured, Velt calls your `get` and `delete` handlers to fetch and remove notification data from your own backend instead of Velt's storage. The resolution pipeline runs notification → user → comment, ensuring resolver-enriched user references are available when the user resolver runs. Only notifications with `notificationSource === 'custom'` are routed through the resolver. [Learn more →](/self-host-data/notifications) +- [**Notifications**]: Added self-hosting options for custom notifications. When configured, Velt calls your `get` and `delete` handlers to fetch and remove notification data from your own backend instead of Velt's storage. The resolution pipeline runs notification → user → comment, ensuring resolver-enriched user references are available when the user resolver runs. Only notifications with `notificationSource === 'custom'` are routed through the resolver. [Learn more →](/self-hosting/partial/notifications) @@ -1013,7 +1013,7 @@ Velt.setDataProviders({ The [`Notification`](/api-reference/sdk/models/data-models#notification) model gains `isNotificationResolverUsed?: boolean` to detect resolver-enriched notifications downstream. -- [**Notifications**] [**API**]: Added server-side schema support for the Notification Resolver. `displayHeadlineMessageTemplate` and `displayBodyMessage` are no longer required when `isNotificationResolverUsed: true` is set on a custom notification write — Velt stores the structural notification and the resolver supplies PII at read time. [Learn more →](/self-host-data/notifications) +- [**Notifications**] [**API**]: Added server-side schema support for the Notification Resolver. `displayHeadlineMessageTemplate` and `displayBodyMessage` are no longer required when `isNotificationResolverUsed: true` is set on a custom notification write — Velt stores the structural notification and the resolver supplies PII at read time. [Learn more →](/self-hosting/partial/notifications) @@ -1143,7 +1143,7 @@ commentElement.getCommentAnnotationCount({ ### New Features -- [**Recorder**]: Added `recorder` field on `VeltDataProvider` accepting a `RecorderAnnotationDataProvider`. When configured via `setDataProviders()`, recording PII (user identity, transcription, attachment URLs) is stripped from annotations before being written to Velt's Firestore and routed to your own storage via resolver callbacks or config-based HTTP endpoints. On read, the resolved PII is merged back transparently. [Learn more →](/self-host-data/recordings) +- [**Recorder**]: Added `recorder` field on `VeltDataProvider` accepting a `RecorderAnnotationDataProvider`. When configured via `setDataProviders()`, recording PII (user identity, transcription, attachment URLs) is stripped from annotations before being written to Velt's Firestore and routed to your own storage via resolver callbacks or config-based HTTP endpoints. On read, the resolved PII is merged back transparently. [Learn more →](/self-hosting/partial/recordings) @@ -1379,7 +1379,7 @@ commentElement.on('visibilityOptionClicked').subscribe((event) => { ### New Features -- [**Comments**]: Added [`AnonymousUserDataProvider`](/self-host-data/users#anonymous-user-resolution) to resolve email → `userId` mappings for users who are tagged by email in comments but are not part of the contact list. When a comment is saved and a tagged contact or `to` recipient has an email but no `userId`, the SDK calls the registered provider to look up the `userId` and backfills it into the comment data before persisting. +- [**Comments**]: Added [`AnonymousUserDataProvider`](/self-hosting/partial/users#anonymous-user-resolution) to resolve email → `userId` mappings for users who are tagged by email in comments but are not part of the contact list. When a comment is saved and a tagged contact or `to` recipient has an email but no `userId`, the SDK calls the registered provider to look up the `userId` and backfills it into the comment data before persisting. diff --git a/release-notes/version-6/sdk-changelog.mdx b/release-notes/version-6/sdk-changelog.mdx index 940f14a40..5ba59022b 100644 --- a/release-notes/version-6/sdk-changelog.mdx +++ b/release-notes/version-6/sdk-changelog.mdx @@ -118,7 +118,7 @@ description: Release Notes of changes added to the core Velt SDK ### Improvements -- [**Self-Hosting**]: New optional `dataRegions` field on the self-hosted config declares which regional data endpoints you provisioned — writes routed to an unprovisioned region now fall back to the base endpoint instead of failing. Adds `asiaNorthEast1` (Tokyo) to the known regions. [Learn more →](/self-host-data/overview) +- [**Self-Hosting**]: New optional `dataRegions` field on the self-hosted config declares which regional data endpoints you provisioned — writes routed to an unprovisioned region now fall back to the base endpoint instead of failing. Adds `asiaNorthEast1` (Tokyo) to the known regions. [Learn more →](/self-hosting/partial/overview) ### Bug Fixes @@ -246,7 +246,7 @@ This release brings the Comment Sidebar V2 to parity with V1 across grouping, fi ### Improvements -- [**Self-Hosting**]: The `fieldsToRemove` allowlist now applies to reaction annotations, recorder annotations, and all activity feature types (previously only comments and `custom` activities). [Learn more →](/self-host-data/overview#excluding--extending-fields) +- [**Self-Hosting**]: The `fieldsToRemove` allowlist now applies to reaction annotations, recorder annotations, and all activity feature types (previously only comments and `custom` activities). [Learn more →](/self-hosting/partial/overview#excluding--extending-fields) - [**Comments**]: Explicitly selecting "Resolved" (or any terminal status) in the V2 sidebar status filter now surfaces those comments instead of hiding them by default. [Learn more →](/async-collaboration/comments-sidebar/v2/customize-behavior) diff --git a/scripts/skills-sync/reference-artifacts/mapping.md b/scripts/skills-sync/reference-artifacts/mapping.md index 39aa1caf4..92bd69f34 100644 --- a/scripts/skills-sync/reference-artifacts/mapping.md +++ b/scripts/skills-sync/reference-artifacts/mapping.md @@ -21,7 +21,7 @@ These take priority over all convention rules. Use for cross-cutting files where | `get-started/**` | `velt-setup-best-practices` | | `security/auth-tokens.mdx`, `security/jwt-tokens.mdx`, `security/supported-regions.mdx` | `velt-setup-best-practices` | | `security/proxy-server.mdx`, `security/content-security-policy.mdx` | `velt-proxy-server-best-practices` | -| `self-host-data/**` | `velt-self-hosting-data-best-practices` | +| `self-hosting/partial/**` | `velt-self-hosting-data-best-practices` | | `backend-sdks/python.mdx` | `velt-self-hosting-data-best-practices` | | `backend-sdks/node.mdx` | `velt-node-sdk-best-practices` | | `api-reference/rest-apis/v1/**`, `api-reference/rest-apis/v2/**` | `velt-rest-apis-best-practices` | diff --git a/self-hosting/full/aws.mdx b/self-hosting/full/aws.mdx new file mode 100644 index 000000000..3b47f29e5 --- /dev/null +++ b/self-hosting/full/aws.mdx @@ -0,0 +1,25 @@ +--- +title: "AWS (Closed Beta)" +sidebarTitle: "AWS" +description: "Velt full self-hosting on AWS is in closed beta. Contact us for access." +--- + + + Full self-hosting on AWS is in **closed beta**. Contact us to request access. + + +Full self-hosting is generally available on GCP plus Firebase today. AWS support is being built out with design partners, so access is granted case by case rather than self-serve. + +## What's available today + +- [Get Started on GCP](/self-hosting/full/gcp/overview) is the generally available path. If your team can deploy into a GCP project, you can run the full stack now. +- [Overview](/self-hosting/full/overview) covers the concepts that apply to every cloud: the umbrella release, the signed manifest, deployment profiles, and strict mode. None of these change per platform, so reading it now is not wasted effort. + +## Request access + +Contact us to join the closed beta. Include the following so we can tell you where you land in the queue: + +- Target cloud and expected region. +- The [deployment profile](/self-hosting/full/gcp/reference#deployment-profiles) you need, and any opt-in modules. +- Rough scale: number of end users, documents, and expected concurrency. +- Whether a GCP deployment would unblock you in the meantime. diff --git a/self-hosting/full/azure.mdx b/self-hosting/full/azure.mdx new file mode 100644 index 000000000..376451403 --- /dev/null +++ b/self-hosting/full/azure.mdx @@ -0,0 +1,25 @@ +--- +title: "Azure (Closed Beta)" +sidebarTitle: "Azure" +description: "Velt full self-hosting on Azure is in closed beta. Contact us for access." +--- + + + Full self-hosting on Azure is in **closed beta**. Contact us to request access. + + +Full self-hosting is generally available on GCP plus Firebase today. Azure support is being built out with design partners, so access is granted case by case rather than self-serve. + +## What's available today + +- [Get Started on GCP](/self-hosting/full/gcp/overview) is the generally available path. If your team can deploy into a GCP project, you can run the full stack now. +- [Overview](/self-hosting/full/overview) covers the concepts that apply to every cloud: the umbrella release, the signed manifest, deployment profiles, and strict mode. None of these change per platform, so reading it now is not wasted effort. + +## Request access + +Contact us to join the closed beta. Include the following so we can tell you where you land in the queue: + +- Target cloud and expected region. +- The [deployment profile](/self-hosting/full/gcp/reference#deployment-profiles) you need, and any opt-in modules. +- Rough scale: number of end users, documents, and expected concurrency. +- Whether a GCP deployment would unblock you in the meantime. diff --git a/self-hosting/full/gcp/install.mdx b/self-hosting/full/gcp/install.mdx new file mode 100644 index 000000000..0980934b7 --- /dev/null +++ b/self-hosting/full/gcp/install.mdx @@ -0,0 +1,1326 @@ +--- +title: "Install Guide" +sidebarTitle: "Install" +description: "Agent-executable procedure for a greenfield Velt full self-hosting deployment on GCP. Covers release verification, project bootstrap, Terraform apply, workspace provisioning, console, SDK hosting, and acceptance." +--- + + + This guide is written to be executed by an AI coding agent with shell access. Hand it the page and let it work. A human can also follow it manually, since every step is a plain command. Want the short version first? Read [Get Started on GCP](/self-hosting/full/gcp/overview). + + +> **Doc set.** Human orientation + concepts: [Overview](/self-hosting/full/overview). +> Field-level contracts (manifest, profiles, configs, trust): [Reference](/self-hosting/full/gcp/reference). +> After install, upgrades: [Upgrade guide](/self-hosting/full/gcp/upgrade). +> This file is the **executable install procedure**. Paste it to an agent, or follow the commands yourself. + +**Audience:** an AI coding agent with shell access, working for a Velt customer. A human +operator pastes this document (or its URL) to you as instructions. A human can also +follow it manually, since every step is a plain command. + +**Outcome:** the complete Velt platform running entirely on the customer's own +infrastructure: + +1. **Backend**: Velt's signed container deployed as Cloud Run services in the + customer's GCP project (Terraform), with a bootstrapped workspace and API keys. +2. **Console**: the Velt admin console served from a static host the customer + chooses (their existing host, a GCP-native stack, or Firebase Hosting), talking + only to their own project. +3. **SDK**: the Velt client SDK served from the customer's own CDN/static host. + +When finished, the customer's app and admin surface make **zero runtime requests to any +Velt-owned host**. + +--- + +## To the AI agent: read this first + +- **Execute phases in order** (Phase 5 may run in parallel with Phase 4). Each phase + ends with a **Verify** block; do not proceed until it passes. +- **Maintain the state file** (`velt-selfhost-state.json`, contract below) after every + step. If your session is interrupted at any point, a fresh session resumes by reading + this guide plus the state file, never by memory. +- **Never run commands against any project other than `$PROJECT_ID`.** +- **All versions come from the release manifest** (Phase 0). Never substitute a + different component version than the manifest pins. +- **Human-in-the-loop moments are known in advance.** Show the operator the table in + Phase 0 §"When you'll be needed" before starting, and when a moment arrives, hand + them the *exact* command or console URL, never a vague ask. +- **Retryable errors:** these are expected and must be retried with backoff, not + reported as failures, (a) Eventarc "Permission denied while using the Eventarc + Service Agent" within ~10 min of enabling the Eventarc API (wait 5 min, re-apply); + (b) 401s mid-`terraform apply` from an expired access token (refresh auth, + re-apply, Terraform state makes it idempotent); (c) `webApps`/service listings + empty seconds after creation (propagation; wait 10 s, re-list); (d) a + just-created service account returning "does not exist" on an IAM binding + (wait ~15 s, retry). +- **Long waits are normal.** The full run is ~2–4 hours of wall clock, most of it + waiting on applies and index creation. Flagged inline. Tell the operator when a + long wait starts so the run doesn't look hung. +- **Shell quirks:** commands are written for bash/zsh. In zsh, `$VAR:something` + is parsed as a parameter modifier and silently corrupts the value, always write + `${VAR}` when a colon (or any word character) follows. When a pipeline's last + command succeeds, earlier failures are masked; check the command you care about, + not just the pipeline's exit code. +- If a command fails and it isn't in the retryable list, check the Troubleshooting + appendix before escalating to the operator. + +### State file contract: `velt-selfhost-state.json` + +Keep it in the working directory (or the operator's config repo). It is the single +source of truth for resume and for cross-phase handoffs. Shape: + +```jsonc +{ + "guideVersion": 1, + "release": { /* the full manifest fetched in Phase 0 */ }, + "inputs": { + "projectId": "", "projectNumber": "", "region": "us-central1", + "profile": "core", "optInModules": [], + "ownerEmail": "", "workspaceName": "", "adminEmails": [], "appDomains": [], + "consoleHost": "", "consoleBase": "", "consoleSiteId": "", + "sdkHost": "", "cdnBase": "" + }, + "phases": { + "0-preflight": "pending|done", + "1-project-bootstrap": "pending|in-progress|done", + "2-backend": "…", "3-workspace": "…", "4-console": "…", + "5-sdk-cdn": "…", "6-acceptance": "…" + }, + "artifacts": { + "moduleDir": "", // where the backend module archive was extracted + "imageRef": "", // customer-registry ref, pinned BY DIGEST + "rtdbUrls": {}, // default + plugin/integrations/demo/notifications + "storageBucket": "", + "cacheddataUrl": "", + "serviceUrls": {}, // terraform output (or its file path) + "bootstrapResultPath": "", // provision-cli JSON — SECRET, path only, never inline + "consoleUrl": "", + "sdkCdnPath": "", // e.g. https://static.acme.com/lib/sdk@6.0.0 + "selfHostedConfigPath": "" // velt-selfhosted-config.json (Phase 5.1) — the app's config.selfHosted object + }, + "humanSteps": { "billing": "pending|done", "oauthClient": "pending|done", "infosecScan": "pending|done|waived", "dnsRecord": "pending|done|n/a", "signInTest": "pending|done" }, + "log": [ { "ts": "…", "phase": "…", "note": "…" } ] +} +``` + +Secrets (API keys, auth tokens, the provision-cli result JSON, SA keys) go in the +operator's secret store. Record **paths/references** in the state file, never values. + +--- + +## Phase 0: Release resolution, inputs, preflight + +### 0.1 Resolve the release + +All release artifacts live in ONE public Artifact Registry repo, the same registry +the container image is pulled from. Non-image artifacts (manifest, module archive, +console bundle) are OCI artifacts fetched with `oras` (anonymous, no auth needed): + +```bash +RELEASE_REGISTRY="us-docker.pkg.dev/velt-sdk/velt-releases" + +# Latest release (or pin a version: velt-selfhost-manifest:) +oras pull -o . "$RELEASE_REGISTRY/velt-selfhost-manifest:latest" # writes manifest.json + +# MANDATORY: verify the manifest's cosign signature — resolve the pulled tag to its +# digest, then verify. Only deploy releases that verify successfully. +MANIFEST_DIGEST=$(oras manifest fetch --descriptor "$RELEASE_REGISTRY/velt-selfhost-manifest:latest" | python3 -c "import json,sys;print(json.load(sys.stdin)['digest'])") +cosign verify \ + --certificate-identity-regexp 'https://github\.com/[^/]+/shared-firebase-function/.*' \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com \ + "$RELEASE_REGISTRY/velt-selfhost-manifest@$MANIFEST_DIGEST" +``` + +The signed manifest is the trust root for everything else: its sha256 fields pin the +module archive and console bundle, and `backend.digest` pins the image (which gets its +own cosign verification in Phase 1). The manifest pins everything you will deploy: + +| Manifest field | Used in | +|---|---| +| `registry` | all `oras` pulls below (same value as `RELEASE_REGISTRY`) | +| `backend.imageByDigest` | Phase 1 (copy + verify), Phase 2 (`velt_image`) | +| `backend.moduleRef` + `backend.moduleSha256` | Phase 2: the Terraform module archive (pull, verify, extract) | +| `console.bundleRef` + `console.sha256` | Phase 4 | +| `sdk.testedVersion` / `sdk.minVersion` + `sdk.bundleRef` + `sdk.sha256` | Phase 5 | +| `releaseNotes` | show the operator | + +Store the whole manifest in the state file. **Verify:** the manifest parses, has +`schemaVersion: 2`, and all fields above are non-empty. If `schemaVersion` is higher +than `2`, re-fetch this guide from the Velt docs site before continuing. + +### 0.2 Inputs to collect from the operator (all up front, once) + +| Input | Meaning | Example | +|---|---|---| +| `PROJECT_ID` | GCP project to deploy into (existing, or a name to create) | `acme-velt` | +| `REGION` | Home region for Cloud Run/queues (`us-central1` recommended; RTDB only exists in us-central1 / europe-west1 / asia-southeast1). On a reused project this MUST match the previous deployment; see the reused-project gate below | `us-central1` | +| `PROFILE` | Backend feature profile: `core` \| `core+recording` \| `core+ai+agents` \| `full` | `core` | +| `OPT_IN_MODULES` | Extra modules (recommend `migrations,ai` for the full console experience) | `migrations,ai` | +| `OWNER_EMAIL` | Workspace owner (console login identity) | `admin@acme.com` | +| `WORKSPACE_NAME` | Company / workspace display name (provision-cli `--workspace-name`, Phase 3) | `Acme Inc` | +| `ADMIN_EMAILS` | Additional console admins (comma-separated, may be empty) | `dev@acme.com` | +| `APP_DOMAINS` | Domains the customer's app runs on (SDK `allowedDomains`) | `app.acme.com` | +| `CONSOLE_HOST` | **Ask, never assume:** where the console SPA will be served from: the operator's existing static host / a GCP-native stack they want provisioned (GCS + LB + CDN, Cloud Run) or a Firebase Hosting site in this project (the zero-extra-infra option; offer it, don't default to it) | `existing host` \| `firebase-hosting` | +| `CONSOLE_BASE` | HTTPS origin the console is served from (follows from `CONSOLE_HOST`; for Firebase Hosting it is `https://.web.app`) | `https://velt-console.acme.com` | +| `CONSOLE_SITE_ID` | Only if `CONSOLE_HOST = firebase-hosting`: the Hosting site id (globally unique) | `acme-velt-console` | +| `SDK_HOST` | **Ask, never assume:** where the SDK files will be served from: the operator's existing CDN/static host (Cloudflare, CloudFront+S3, Fastly, nginx, …) or, if they have none, a Firebase Hosting site in this project (offer it as the zero-extra-accounts option, not the default) | `existing CDN` \| `firebase-hosting` | +| `CDN_BASE` | HTTPS origin the SDK will be served from (follows from `SDK_HOST`; for Firebase Hosting it is `https://.web.app`) | `https://static.acme.com` | +| `GOOGLE_GEN_AI_API_KEY` | **MANDATORY**: the operator's own Gemini API key (memory embeddings + knowledge search run on Gemini even on core profiles; a placeholder fails at runtime with API_KEY_INVALID). https://aistudio.google.com/apikey | `AIza…` | +| `ANTHROPIC_API_KEY` | **MANDATORY**: the operator's own Anthropic API key (AI/agent execution paths). https://console.anthropic.com/settings/keys | `sk-ant-…` | + +`CONSOLE_HOST` and `SDK_HOST` are explicit operator decisions. Both artifacts are +plain static files; any host meeting the phase's serving rules works (Phase 4.2 for +the console, Phase 5 for the SDK), and most customers have a static host they'd +prefer over a new Firebase site. Present the choice; only pick for them if they say +"you choose". The console has one extra serving requirement the SDK doesn't: it is a +SPA, so unknown paths must fall back to `index.html` (Phase 4.2). + +**DNS feasibility check (do this NOW, not in Phase 5):** if `CDN_BASE` or +`CONSOLE_BASE` uses a custom hostname (anything that isn't `*.web.app` or an +already-live host), ask the operator explicitly: *"Do you control the DNS zone for +``, and can you (or someone reachable now) create an A/CNAME record within +the hour?"* A hostname in a zone the operator can't touch (e.g. the company's +production apex zone managed by another team) is the single biggest wall-clock risk +in this guide. If the answer is no or unsure, +resolve the hostname choice **before** starting Phase 1; a Google-managed cert cannot +go ACTIVE until the DNS record is visible. + +**Reused-project gate (do this NOW if `PROJECT_ID` has hosted a Velt deployment +before, even one that was fully `terraform destroy`ed):** a previously-deployed +project carries two invisible constraints that will otherwise fail Phase 2 hours in: + +1. **Cloud Tasks queue-name tombstones.** Deleting a queue blocks re-creating a + queue with the same name in the same region, Terraform fails with + `FAILED_PRECONDITION: … existed too recently`. Google documents ~7 days; + observed sometimes shorter (~3 days), never assume less than 7. +2. **Cloud Run regions-per-project cap, and destroy does NOT release it.** The + default `setencrypteddata_regions = null` layout pins 4 extra regions + (europe-west1, asia-southeast1, australia-southeast1, asia-northeast1); with the + home region that is 5: exactly the default Cloud Run per-project region cap, + zero headroom. Region *initialization* survives a destroy, so switching the home + region on a reused project asks for a 6th region and every service fails with + `Project failed to initialize in this region due to quota exceeded` / + `Resource readiness deadline exceeded`. **Switching regions is never the escape + route from a tombstone.** + +Decision rule: **prefer a fresh project** for any re-run within ~7 days of a +destroy. If the project must be reused, keep `REGION` identical to the previous +deployment and either wait out the tombstone window or accept that the Phase 2 +queue-create step may fail until it clears. If a new home region is genuinely +required, that is a Cloud Run quota-increase request (regions per project) plus the +operator's sign-off, not a tfvars tweak. + +### 0.3 When you'll be needed (show the operator this table now) + +| # | Moment | What | Time | +|---|---|---|---| +| 1 | Phase 1, start | **Billing**: link a billing account to the project (if the deploying identity lacks `roles/billing.user`, a billing admin runs one link command) | 2–5 min | +| 2 | **Now** (used in Phase 4) | **OAuth client**: create one OAuth 2.0 Web client in the Cloud console and paste its id/secret (Google does not allow creating OAuth clients via API) | 3 min | +| 3 | Phase 1.5 | **Infosec scan sign-off**: if org policy requires image scanning, review the scan findings against the signed manifest's known-findings list and approve (or waive scanning explicitly) | 5–15 min | +| 4 | Phase 6 | **Sign-in test**: sign in to the console and the demo app in a browser | 5 min | +| - | If a custom hostname is used (Phase 4/5) | **DNS record**: create the A/CNAME record for the console/SDK hostname the moment the IP is handed over (feasibility confirmed in 0.2; the managed cert waits on this) | 2 min + propagation | +| - | If AI modules enabled | Provide LLM API keys (OpenAI/Anthropic/Google) as secret values | 2 min | + +**Issue the OAuth ask (moment #2) right now, at Phase 0.** Every value it needs is +already known (`PROJECT_ID`, `CONSOLE_BASE`), the exact wording is in Phase 4.3, +and the operator can create the client while you deploy. Collecting it up front turns +what is otherwise the single largest idle wait of the run (deployment done, waiting on +a human) into a parallel task. Note: the project must exist before the operator can +open the credentials page, if you are creating the project in Phase 1, issue the ask +immediately after `gcloud projects create`. + +Everything else is automated. Expected total wall clock: **2–5 hours** (≈ 15 min +Terraform applies, ≈ 10 min–2.5 h workspace bootstrap, index creation dominates and +varies run to run; the rest verification and waiting). +Expected idle cost of the deployed stack: Cloud Run scales per catalog min-instances; +Secret Manager/Artifact Registry are cents per month. Ask the operator whether this is +a production deployment (keep catalog min-instances) or an evaluation +(`min_instances_override = 0` → near-$0 idle). + +### 0.4 Tooling preflight (hard requirements: check ALL before starting) + +| Tool | Why | Check | +|---|---|---| +| Node.js ≥ 20 | deployment-profiles CLI under Terraform | `node --version` | +| Docker daemon | provision-cli container, image copy | `docker info` | +| terraform CLI | the deployment | `terraform version` (install: `brew install hashicorp/tap/terraform`, not in homebrew core) | +| gcloud CLI | all provisioning | see auth probes below | +| firebase CLI (only if a Firebase Hosting host was chosen in 0.2) | Hosting deploy (Phase 4/5) | `firebase --version` | +| cosign | manifest + image signature verification (MANDATORY) | `cosign version` | +| oras | pulling release artifacts from the OCI registry (Phase 0/2/4) | `oras version` (install: `brew install oras`) | +| crane (optional, recommended) | digest-preserving image copy in Phase 1.4 without pulling multi-GB layers locally | `crane version` (install: `brew install crane`; docker fallback documented in 1.4) | +| curl, python3, npm | verification, SDK packaging | - | + +(Install hints show Homebrew; on Linux/CI use your package manager or each tool's +official installer; any current version works.) + +**Auth probes: a listed account is NOT proof of working auth, so probe with real calls:** + +```bash +gcloud projects list --limit=1 # fails on stale token → gcloud auth login +gcloud auth application-default print-access-token >/dev/null \ + || echo "ADC MISSING → gcloud auth application-default login (Terraform needs ADC, NOT the CLI token)" +firebase projects:list >/dev/null 2>&1 \ + || echo "firebase CLI has its OWN token store → firebase login --reauth" +``` + +Non-interactive fallback if ADC can't be refreshed right now: +`export GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token)`, but tokens live +~60 min and a long apply can outlive one; prefer real ADC. + +**Verify (phase gate):** every tool present; all three auth probes pass; manifest +stored; inputs recorded in the state file; operator has acknowledged the +human-moments table. Mark `0-preflight: done`. + +--- + +## Phase 1: GCP project bootstrap + +Everything here is project-lifecycle infrastructure that the Terraform module +deliberately does not model. + +### 1.1 Project + billing (human moment #1) + +```bash +gcloud projects create "$PROJECT_ID" # skip if using an existing project +PROJECT_NUMBER=$(gcloud projects describe "$PROJECT_ID" --format="value(projectNumber)") # → state file +``` + +Billing must be linked before most APIs can be enabled. Test permission first +(side-effect-free), then act or escalate precisely: + +```bash +gcloud billing accounts list # if empty/no permission → operator ask below +gcloud billing projects link "$PROJECT_ID" --billing-account="$BILLING_ACCOUNT_ID" +``` + +> **Operator ask (verbatim, if the link fails):** "I need billing linked to +> `$PROJECT_ID`. Either grant me `roles/billing.user` on your billing account, or have +> a billing admin run: +> `gcloud billing projects link $PROJECT_ID --billing-account=`" + +### 1.2 Firebase + APIs + +```bash +TOKEN=$(gcloud auth print-access-token) +AUTH=(-H "Authorization: Bearer $TOKEN" -H "X-Goog-User-Project: $PROJECT_ID" -H "Content-Type: application/json") +# The X-Goog-User-Project header is MANDATORY on every firebase*/identitytoolkit REST +# call made with user credentials (403 SERVICE_DISABLED without it). +# Content-Type is MANDATORY on every POST carrying a -d body. Harmless on GETs. + +# Enable APIs FIRST — addFirebase below needs firebase.googleapis.com already on. +gcloud services enable --project="$PROJECT_ID" \ + firebase.googleapis.com firestore.googleapis.com firebasedatabase.googleapis.com \ + firebaserules.googleapis.com identitytoolkit.googleapis.com eventarc.googleapis.com \ + iam.googleapis.com run.googleapis.com cloudscheduler.googleapis.com \ + cloudtasks.googleapis.com secretmanager.googleapis.com artifactregistry.googleapis.com \ + firebasestorage.googleapis.com storage.googleapis.com pubsub.googleapis.com + +# Attach Firebase (REST — works even when the firebase CLI token is stale). +# ⚠ ALWAYS brace variables followed by a colon: in zsh, $PROJECT_ID:addFirebase +# triggers the `:a` (absolute-path) parameter modifier and silently mangles the URL +# into an HTML-404-producing path. ${PROJECT_ID}: is safe. +# ⚠ If the project-ID form errors, retry with the PROJECT NUMBER — on fresh projects +# the ID form has been seen to fail where the number form succeeds. +curl -sf -X POST "${AUTH[@]}" \ + "https://firebase.googleapis.com/v1beta1/projects/${PROJECT_ID}:addFirebase" \ +|| curl -sf -X POST "${AUTH[@]}" \ + "https://firebase.googleapis.com/v1beta1/projects/${PROJECT_NUMBER}:addFirebase" + +# addFirebase is async — poll until state=ACTIVE before continuing (≤ ~2 min): +until curl -s "${AUTH[@]}" "https://firebase.googleapis.com/v1beta1/projects/$PROJECT_ID" \ + | python3 -c "import json,sys; exit(0 if json.load(sys.stdin).get('state')=='ACTIVE' else 1)"; do + sleep 10 +done +``` + +### 1.3 Firestore, Realtime Databases, Auth, default bucket + +```bash +# Firestore native — LOCATION IS IRREVERSIBLE. nam5 (US multi-region) / eur3 (EU). +gcloud firestore databases create --project="$PROJECT_ID" \ + --location=nam5 --type=firestore-native +# Record firestore_location in the state file — tfvars needs it to match EXACTLY. + +# Default RTDB instance +curl -sf -X POST "${AUTH[@]}" -H "Content-Type: application/json" \ + "https://firebasedatabase.googleapis.com/v1beta/projects/$PROJECT_ID/locations/$REGION/instances?databaseId=$PROJECT_ID-default-rtdb" \ + -d '{"type":"DEFAULT_DATABASE"}' + +# Four additional RTDB instances (plugin / integrations / demo / notifications) +for name in plugin integrations demo notifications; do + curl -sf -X POST "${AUTH[@]}" -H "Content-Type: application/json" \ + "https://firebasedatabase.googleapis.com/v1beta/projects/$PROJECT_ID/locations/$REGION/instances?databaseId=$PROJECT_ID-$name" \ + -d '{"type":"USER_DATABASE"}' +done +# Record all five URLs (https://.firebaseio.com) in the state file. + +# Firebase Auth (Identity Platform) +curl -sf -X POST "${AUTH[@]}" \ + "https://identitytoolkit.googleapis.com/v2/projects/$PROJECT_ID/identityPlatform:initializeAuth" -d '{}' + +# ⚠ VERIFY it actually initialized — `curl -sf` swallows failures, and a silently +# failed initializeAuth surfaces much later as a confusing Phase 2.3 import failure +#. Expect HTTP 200 with a config JSON: +curl -s -o /dev/null -w "%{http_code}\n" "${AUTH[@]}" \ + "https://identitytoolkit.googleapis.com/admin/v2/projects/$PROJECT_ID/config" +# 200 → initialized. 404 (CONFIGURATION_NOT_FOUND) → NOT initialized; re-run +# initializeAuth. Record the outcome — Phase 2.3's import step is conditional on it. + +# Default storage bucket. ⚠ New projects get .firebasestorage.app, +# NOT .appspot.com — record the ACTUAL name; tfvars must set it explicitly. +curl -sf -X POST "${AUTH[@]}" \ + "https://firebasestorage.googleapis.com/v1beta/projects/$PROJECT_ID/defaultBucket" + +# ⚠ FALLBACK: on some fresh projects the POST above fails on every +# variant tried — seen as both 404s AND 400 INVALID_ARGUMENT. Creating App Engine +# provisions the default bucket instead (as .appspot.com — App Engine region +# choice is PERMANENT, match $REGION): +# gcloud app create --region=us-central --project="$PROJECT_ID" +# ⚠ The App-Engine-created bucket is NOT automatically linked to Firebase Storage — +# the GET on defaultBucket keeps 404ing until you link it explicitly: +# curl -sf -X POST "${AUTH[@]}" \ +# "https://firebasestorage.googleapis.com/v1beta/projects/$PROJECT_ID/buckets/${PROJECT_ID}.appspot.com:addFirebase" -d '{}' +# Then confirm with a GET on defaultBucket and record the ACTUAL bucket name. +``` + +### 1.4 Verify + copy the signed image + +Verify **against Velt's registry, before copying** (signatures attach to the digest in +the source repo), then copy by digest into the customer's own Artifact Registry: + +```bash +IMAGE_BY_DIGEST=$(python3 -c "import json;print(json.load(open('manifest.json'))['backend']['imageByDigest'])") + +cosign verify \ + --certificate-identity-regexp 'https://github.com/.*/shared-firebase-function/\.github/workflows/container-release\.yml@.*' \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com \ + "$IMAGE_BY_DIGEST" +# MUST print Verified OK. If it fails, STOP — do not deploy an unverifiable image. + +gcloud artifacts repositories create velt --project="$PROJECT_ID" \ + --repository-format=docker --location="$REGION" +gcloud auth configure-docker "$REGION-docker.pkg.dev" --quiet + +LOCAL_IMAGE="$REGION-docker.pkg.dev/$PROJECT_ID/velt/velt-functions:$(python3 -c "import json;print(json.load(open('manifest.json'))['selfHostVersion'])")" + +# Preferred: registry-to-registry copy — digest-preserving, no multi-GB round-trip +# through the local machine, no architecture pitfalls (`brew install crane`): +crane copy "$IMAGE_BY_DIGEST" "$LOCAL_IMAGE" + +# Fallback (no crane): docker pull/tag/push. ⚠ The image is linux/amd64-only — on an +# Apple Silicon / arm64 host a bare `docker pull ` fails with "no matching +# manifest for linux/arm64". Always pass the platform explicitly: +docker pull --platform linux/amd64 "$IMAGE_BY_DIGEST" +docker tag "$IMAGE_BY_DIGEST" "$LOCAL_IMAGE" +docker push "$LOCAL_IMAGE" + +# Confirm the pushed digest matches the manifest digest, then pin BY DIGEST: +gcloud artifacts docker images describe "$LOCAL_IMAGE" --format="value(image_summary.digest)" +``` + +Record `artifacts.imageRef` in the state file as +`@`. Terraform gets the digest form, never the tag. + +### 1.5 Vulnerability-scan the image copy (operator's infosec policy) + +**Ask the operator whether their organization requires vulnerability scanning of +third-party images before deployment** (most enterprises do; the AWS equivalent is +ECR scan-on-push). If yes, run it now against THEIR registry copy and get their +sign-off before Phase 2; a finding dispute discovered mid-deploy is far more +expensive than one found here. + +GCP-native flow (Artifact Analysis): either enable auto-scan-on-push in the +customer project (each pushed image version is scanned automatically, ~$0.26/image): + +```bash +gcloud services enable containerscanning.googleapis.com --project="$PROJECT_ID" +# then push (1.4) and read results once analysis completes: +gcloud artifacts docker images list-vulnerabilities "$LOCAL_IMAGE" --project="$PROJECT_ID" +``` + +or run a one-shot on-demand scan (no per-project API cost surprises; ~2–4 min on +this image): + +```bash +gcloud services enable ondemandscanning.googleapis.com --project="$PROJECT_ID" +SCAN=$(gcloud artifacts docker images scan "$LOCAL_IMAGE" --remote \ + --project="$PROJECT_ID" --quiet --format="value(response.scan)") +# --quiet matters: the first run installs gcloud's local-extract component and +# otherwise hangs forever on an interactive Y/n prompt. +gcloud artifacts docker images list-vulnerabilities "$SCAN" --project="$PROJECT_ID" +``` + +If the org routes all third-party images through a central scanner (ECR, Harbor, +Prisma, …), hand their infosec team the image digest plus Velt's signed SBOM; it is +attached to the image as a cosign SPDX attestation and verifiable offline: + +```bash +cosign verify-attestation --type spdxjson \ + --certificate-identity-regexp 'https://github.com/.*/shared-firebase-function/\.github/workflows/container-release\.yml@.*' \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com \ + "$IMAGE_BY_DIGEST" | python3 -c "import json,sys,base64;print(base64.b64decode(json.load(sys.stdin)['payload']).decode())" > velt-sbom.spdx.json +``` + +**Setting expectations (show the operator):** a full Node.js runtime image will +never scan clean. Findings cluster as (a) Debian base-OS packages with **no fix +available** upstream (the majority of CRITICAL/HIGH findings are this class); +(b) npm/application findings, Velt's release gate blocks on fixable CRITICAL/HIGH. +**Decision rule:** compare your scan's CRITICAL/HIGH findings against the release's +known-findings list, which ships inside the **signed manifest** as +`backend.knownFindings` (each entry: `{id, package, severity, fixAvailable}`): + +```bash +python3 -c "import json;print(json.dumps(json.load(open('manifest.json'))['backend'].get('knownFindings','ABSENT'),indent=2))" +``` + +Findings in that list = accepted risk, proceed on operator sign-off; a CRITICAL/HIGH +NOT in the list = stop and contact Velt before deploying. If `knownFindings` is +missing or empty, fall back to the manifest's `releaseNotes` URL; if that is also +unavailable, present the raw findings to the operator for judgment (unfixable +Debian base-OS packages are the expected class) and record their explicit decision. +Record the scan result reference and the operator's decision in the state file +(`humanSteps.infosecScan: done`). + +**Verify (phase gate):** Firestore ACTIVE; 5 RTDB instances ACTIVE +(`curl -sf "${AUTH[@]}" https://firebasedatabase.googleapis.com/v1beta/projects/$PROJECT_ID/locations/-/instances`); +Identity Platform probe outcome recorded (200 vs 404, drives the Phase 2.3 import); +default bucket name recorded; cosign Verified OK captured; pushed digest == +manifest digest; infosec scan done + signed off (or operator explicitly waived it). +Mark `1-project-bootstrap: done`. + +--- + +## Phase 2: Backend deployment (Terraform) + +### 2.1 Download + verify the backend module archive + +The manifest's `backend.moduleRef` is a self-contained archive: `terraform/` (the +deployment blueprint; its `README.md` is the authoritative variable reference) and the +precompiled `functions/lib/` (the deployment-profiles CLI that Terraform runs; no npm +install or build needed, only Node ≥ 20). Always use this guide from the Velt docs +site (or your current copy), not any snapshot that may ship inside an older archive. + +```bash +MODULE_REF=$(python3 -c "import json;print(json.load(open('manifest.json'))['backend']['moduleRef'])") +MODULE_SHA=$(python3 -c "import json;print(json.load(open('manifest.json'))['backend']['moduleSha256'])") +oras pull -o . "$MODULE_REF" # writes velt-backend-module-.tar.gz +MODULE_TAR=$(ls velt-backend-module-*.tar.gz) +echo "$MODULE_SHA $MODULE_TAR" | shasum -a 256 -c - # MUST print OK — hard-fail otherwise +export MODULE_DIR="$(pwd)/velt-backend-module" # exported: Phase 5.1's config generator reads it from the environment +mkdir -p "$MODULE_DIR" && tar -xzf "$MODULE_TAR" -C "$MODULE_DIR" +cd "$MODULE_DIR/terraform" +# Record artifacts.moduleDir in the state file. +``` + +⚠ **Never run Terraform in a directory that already carries state from another +install** (a reused module dir, or a checked-out source tree that may hold live +state for a different project). Foreign state turns a fresh install into a +destroy-and-recreate plan against the other project. Always extract into a fresh +`MODULE_DIR`; if `terraform plan` shows ANY destroys on a fresh project, stop , +you are on the wrong state. + +### 2.2 tfvars + +Write `velt.auto.tfvars` (keep it in the operator's config repo): + +```hcl +project_id = "" +region = "" +velt_image = "" +profile = "" +opt_in_modules = [] + +firebase_database_url = "https://-default-rtdb.firebaseio.com" +velt_plugin_database_url = "https://-plugin.firebaseio.com" +velt_integrations_database_url = "https://-integrations.firebaseio.com" +velt_demo_database_url = "https://-demo.firebaseio.com" +velt_notifications_db_instance = "-notifications" +velt_cacheddata_url = "https://pending.invalid" # bootstrap — fed back in 2.5 + +# THIS project's Firebase WEB API key (MANDATORY — the identitytoolkit +# signInWithCustomToken exchange must hit this deployment's own Firebase Auth). +# Fetch after Phase 1 enabled Firebase. NOTE: `api-keys list` does NOT populate +# keyString — list the resource name, then +# get-key-string it: +# KEY_NAME=$(gcloud services api-keys list --project="$PROJECT_ID" \ +# --filter="displayName:'Browser key'" --format='value(name)' | head -1) +# gcloud services api-keys get-key-string "$KEY_NAME" --format='value(keyString)' +# (any of the project's web API keys works — they are project-level; expect "AIza…") +velt_firebase_web_api_key = "" + +firestore_location = "nam5" # MUST match the Firestore DB location +rtdb_location = "" # MUST match the RTDB instances' region +# (rtdb_location also feeds VELT_RTDB_LOCATION at runtime: non-us-central1 RTDB +# instances are only reachable at ..firebasedatabase.app, and +# the backend builds notification-hub URLs from this region.) +firebase_storage_bucket = ".firebasestorage.app>" + +# Console (frontend infra; the console backend functions always deploy) +# velt_portal_url = the console's serving origin (CONSOLE_BASE from Phase 0.2) — +# it drives CORS origin admission, magic-link URLs, and Identity Platform +# authorized_domains. console_hosting_site_id only when CONSOLE_HOST=firebase-hosting +# (leave "" for any other host; the module skips the Hosting site resource). +# TIMING: if the operator has not yet answered CONSOLE_HOST/CONSOLE_BASE (Phase +# 0.2), ASK NOW — before writing this file. Do not write a placeholder and defer +# the question to the Phase 2.4 key-collection stop: velt_portal_url feeds env +# vars on every service, so changing it after apply forces a broad re-apply +#. +console_hosting_site_id = "" +velt_portal_url = "" +console_manage_identity_platform = true +console_database_url = "https://-sdktest.firebaseio.com" # shared testing-key infra; +console_storage_url = "gs://-sdktest" # provision-cli creates these in Phase 3 + +# min_instances_override = 0 # EVALUATION ONLY — scale-to-zero, ~$0 idle +``` + +### 2.3 Static gates + +```bash +terraform init +terraform validate +terraform plan # review: service count follows the profile; NO resources outside $PROJECT_ID +``` + +If the plan fails naming a missing `VELT_*` variable or an unknown profile, that is the +the module validates fail-closed. Fix the tfvars, do not work around it. + +Export these before ANY apply: the firebaserules provider calls route their quota +through the billing project, and the later apply 403s without them: + +```bash +export USER_PROJECT_OVERRIDE=true GOOGLE_BILLING_PROJECT="$PROJECT_ID" +``` + +⚠ **Identity Platform import (CONDITIONAL, key off the Phase 1.3 verify probe):** +if the probe returned 200 (Auth initialized), the singleton config already exists and +MUST be imported (creating it fails ALREADY_EXISTS). If it returned 404 +(`CONFIGURATION_NOT_FOUND`: initializeAuth never took), SKIP the import. The module +creates the config from scratch, and importing a non-existent config fails: + +```bash +terraform import 'google_identity_platform_config.console[0]' "$PROJECT_ID" +``` + +### 2.4 Three-pass apply + +Cloud Run refuses a revision that mounts a secret with no enabled version, so: +containers first, seed, then services. + +```bash +# PASS 1 — secret containers + IAM + Velt-only placeholders (~1 min) +terraform apply -target=google_secret_manager_secret.required \ + -target=google_secret_manager_secret_iam_member.runtime_accessor \ + -target=google_secret_manager_secret_version.optional_placeholder + +# PASS 2 — seed every id in the secrets_to_seed output with a value, out-of-band +# (secret material never enters Terraform state). +# ⚠ THE OUTPUT LIST IS AUTHORITATIVE — the per-key notes below are illustrative and +# vary by module version; seed exactly what the output lists, nothing more or less. +terraform output -json secrets_to_seed +# Value rules by key (apply to whichever keys the output actually lists): +# PLUGIN_CRYPTO_KEY : exactly 32 chars → openssl rand -hex 16 +# PLUGIN_CRYPTO_IV : exactly 16 chars → openssl rand -hex 8 +# JWT_SECRET_KEY : per-deployment → openssl rand -base64 32 (NEVER share across installs) +# GOOGLE_GEN_AI_API_KEY : MANDATORY — a REAL Gemini key from the operator. +# Memory embeddings (knowledge indexing + /memory search) run on Gemini even on +# core/rest-api profiles; a placeholder fails at runtime with API_KEY_INVALID +# . Get one: https://aistudio.google.com/apikey +# ANTHROPIC_API_KEY : MANDATORY — a REAL Anthropic key from the operator +# (AI/agent execution paths). Get one: https://console.anthropic.com/settings/keys +# OPEN_AI_API_KEY : +# - ai/agents/recorder modules ENABLED → real key from the operator +# - modules NOT enabled but the key is still in the list (seen on 0.9.1 core) → +# seed a placeholder string (e.g. "placeholder-not-configured"). Cloud Run +# refuses to mount a secret with no enabled version, so listed = must be seeded +# even if no code path reads it. Do NOT invent real-looking values. +# SVIX_API_KEY / SVIX_API_KEY_US : N/A on self-host — advanced webhooks (the +# Svix-backed WebhookV2 feature) are not offered on self-hosted deployments. +# If listed by the output, seed a placeholder; do NOT provision a Svix account. +openssl rand -base64 32| tr -d '\n' | gcloud secrets versions add JWT_SECRET_KEY --project="$PROJECT_ID" --data-file=- +# (seed every remaining secrets_to_seed id the same way, per the value rules above) + +# PASS 3 — full apply (~10–15 min for a core profile; tell the operator) +terraform apply +``` + +Expected retryables during pass 3: the Eventarc service-agent propagation error +(wait ~5 min, re-apply) and expired-token 401s (refresh, re-apply). Both converge. + +### 2.5 Close the cacheddata bootstrap loop + +```bash +terraform output -json service_urls | python3 -c "import json,sys;print(json.load(sys.stdin)['cacheddata'])" +# → set velt_cacheddata_url in tfvars to this URL, then: +terraform apply # picks the env change up on every service (~5–10 min) +terraform plan # MUST end: "No changes. Your infrastructure matches the configuration." +``` + +Save `terraform output -json service_urls > service-urls.json` and record the path in +the state file (Phase 4 and the final report use it). + +### 2.6 Backend smoke tests + +```bash +VALIDATE_URL=$(python3 -c "import json;print(json.load(open('service-urls.json'))['validateclient'])") +CACHED_URL=$(python3 -c "import json;print(json.load(open('service-urls.json'))['cacheddata'])") + +# callable envelope (inner app-level error is EXPECTED — no workspace exists yet) +curl -s -X POST "$VALIDATE_URL" -H 'Content-Type: application/json' -d '{"data":{"apiKey":"smoke"}}' +# → HTTP 200 {"result": ...} + +# http kind — express routing (400 INVALID_ARGUMENT expected for an empty payload) +curl -s -o /dev/null -w "%{http_code}\n" -X POST "$CACHED_URL" -H 'Content-Type: application/json' -d '{}' +# → 400 +``` + +**Verify (phase gate):** all services Ready=True +(`gcloud run services list --project="$PROJECT_ID" | grep -c False` → 0); final plan +converged ("No changes"); both smokes return the expected envelopes; +`velt-console-config.json` + `console-firebase.json` were emitted next to the module. +Mark `2-backend: done`. + +--- + +## Phase 3: First-workspace bootstrap (provision-cli) + +A freshly deployed backend has no owner. This CLI mints the first workspace, a +testing API key, and the first production API key **with its real per-workspace +infrastructure**. Run it ONCE. + +```bash +# Short-lived key for the runtime SA (delete immediately after — step below). +# ⚠ Host path: use a directory Docker actually shares into its VM (macOS Docker +# Desktop / colima do NOT share host /tmp by default — the mount silently arrives +# empty and the CLI fails with "SA key not found"). $HOME/velt-provision-keys is safe. +KEYDIR="$HOME/velt-provision-keys" && mkdir -p "$KEYDIR" +gcloud iam service-accounts keys create "$KEYDIR/sa.json" \ + --iam-account="velt-functions-runtime@$PROJECT_ID.iam.gserviceaccount.com" --project="$PROJECT_ID" + +# --entrypoint node is REQUIRED: the image's default entrypoint is the +# functions-framework launcher, which fail-louds on a missing FUNCTION_TARGET +# before your command ever runs. +# --platform linux/amd64 is REQUIRED on arm64 hosts (Apple Silicon): the image is +# amd64-only; without the flag the run fails at pull. Emulated execution is fine — +# the CLI is I/O-bound. +docker run --rm --entrypoint node --platform linux/amd64 \ + -e GCLOUD_PROJECT="$PROJECT_ID" \ + -e PROJECT_NUMBER="" \ + -e FIREBASE_CONFIG="{\"projectId\":\"$PROJECT_ID\",\"databaseURL\":\"https://$PROJECT_ID-default-rtdb.firebaseio.com\",\"storageBucket\":\"\"}" \ + -e VELT_PLUGIN_DATABASE_URL="https://$PROJECT_ID-plugin.firebaseio.com" \ + -e VELT_INTEGRATIONS_DATABASE_URL="https://$PROJECT_ID-integrations.firebaseio.com" \ + -e VELT_DEMO_DATABASE_URL="https://$PROJECT_ID-demo.firebaseio.com" \ + -e VELT_NOTIFICATIONS_DB_INSTANCE="$PROJECT_ID-notifications" \ + -e VELT_CACHEDDATA_URL="" \ + -e VELT_HAS_PRODUCTION=true \ + -e CONSOLE_DATABASE_URL="https://$PROJECT_ID-sdktest.firebaseio.com" \ + -e CONSOLE_STORAGE_URL="gs://$PROJECT_ID-sdktest" \ + -e GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json \ + -v "$KEYDIR/sa.json":/keys/sa.json:ro \ + "" \ + lib/deployment-profiles/provision-cli.js \ + --owner-email "" \ + --workspace-name "" \ + --admin-emails "" \ + --allowed-domains "" \ + > bootstrap-result.json +# stdout carries ONLY the result JSON (progress goes to stderr) — the redirect captures it. + +# CLEAN UP THE KEY (both sides) as soon as the run ends: +gcloud iam service-accounts keys list --iam-account="velt-functions-runtime@$PROJECT_ID.iam.gserviceaccount.com" --project="$PROJECT_ID" +gcloud iam service-accounts keys delete --iam-account="velt-functions-runtime@$PROJECT_ID.iam.gserviceaccount.com" --project="$PROJECT_ID" --quiet +rm -rf "$KEYDIR" +``` + +Operational notes: + +- **Mount the SA key under a dedicated path like `/keys`**: mounting under `/tmp` + inside the container fails oddly (and see the host-side `/tmp` sharing warning + above, both sides of the mount have `/tmp` traps). +- **Expect a long "silent but busy" stretch** after progress stops: composite-index + creation for the new databases runs in-process and the CLI exits only when it + drains. Wall clock is typically tens of minutes and can exceed two hours , + index build time varies with Firestore load and is not a hang. Tell the operator + up front; do not kill it. (Poll `gcloud firestore operations list --database=` + in another shell if you want visible progress.) +- The result JSON goes to stdout, but **stray log lines may leak into it**, so do not + assume the captured file is pure JSON. Extract the LAST top-level JSON object + (brace-match from the final `"schemaVersion": 1` occurrence) into + `bootstrap-result.json` before parsing. Its exact shape (`schemaVersion: 1`): + +```jsonc +{ + "schemaVersion": 1, + "workspaceId": "…", // with workspaceAuthToken = the x-velt-workspace-id / + "workspaceAuthToken": "…", // x-velt-auth-token REST header pair + "testingApiKey": "…", + "testingAuthToken": "…", // testing key auth token (API tests / console playground) + "productionApiKey": "…", // the key the customer's app uses + "productionAuthToken": "…", + "firebaseConfig": { … }, // the production key's provisioned infra + "planInfo": { … }, + "adminEmails": [ … ], + "productionStoreDb": { // converge report — exit 0 implies both flags true + "storeDbId": "…", + "metadataSeeded": true, + "compositeIndexesConverged": true + }, + "sharedTestInfra": { "databaseUrl": "…", "storageBucket": "…", "storeDbId": "…" } // absent with --skip-test-infra +} +``` + + **Move it to the operator's secret store**; record only its location in the state + file. + +**Verify (phase gate):** exit 0; result JSON has non-empty workspace id, testing +key, and production key; `productionStoreDb.compositeIndexesConverged` and +`productionStoreDb.metadataSeeded` are both `true` when present. **Still verify +the production DB's composite indexes and apiKey metadata doc below**. Treat +those checks as defense-in-depth even when the CLI reports converge success. + +Confirm the production store DB's composite-index count matches the `sdktest` +database (the CLI awaits and fail-louds the sdktest pass; use it as ground truth): + +```bash +STORE_DB=$(python3 -c "import json;print(json.load(open('bootstrap-result.json'))['firebaseConfig']['storeDbId'])") +EXPECTED=$(gcloud firestore indexes composite list --project="$PROJECT_ID" --database=sdktest --format="value(name)" | wc -l) +ACTUAL=$(gcloud firestore indexes composite list --project="$PROJECT_ID" --database="$STORE_DB" --format="value(name)" | wc -l) +echo "prod DB $ACTUAL / expected $EXPECTED" +# If ACTUAL < EXPECTED, re-run the pass via the deployed handler (idempotent — +# existing indexes 409 as success; takes ~10 s per index, so ~20-25 min for a full pass): +curl -s -X POST "$(python3 -c "import json;print(json.load(open('service-urls.json'))['documentmigrationhandler'])")" \ + -H 'Content-Type: application/json' \ + -d "{\"eventType\":\"create_firestore_indexes\",\"dbId\":\"$STORE_DB\"}" +# → 200 when complete; 500 means a create failed — re-run until 200, then re-count. +``` + +**Also verify the apiKey metadata doc exists in the production store DB.** Security +rules `get()` this doc to resolve the effective access type; a missing doc makes +the rules deny everything (SDK: "Documents provided are all denied"): + +```bash +PROD_KEY=$(python3 -c "import json;print(json.load(open('bootstrap-result.json'))['productionApiKey'])") +TOKEN=$(gcloud auth print-access-token) +DOC_URL="https://firestore.googleapis.com/v1/projects/$PROJECT_ID/databases/$STORE_DB/documents/apiKey/$PROD_KEY" +curl -s "$DOC_URL" -H "Authorization: Bearer $TOKEN" +# → must return fields.metadata.defaultDocumentAccessType = "public". +# If it 404s, create it (byte-identical to what provisioning should have written): +curl -s -X PATCH "$DOC_URL?updateMask.fieldPaths=metadata" -H "Authorization: Bearer $TOKEN" \ + -H 'Content-Type: application/json' \ + -d '{"fields":{"metadata":{"mapValue":{"fields":{"defaultDocumentAccessType":{"stringValue":"public"}}}}}}' +``` + +Then prove the mint end-to-end: + +```bash +PROD_KEY=$(python3 -c "import json;print(json.load(open('bootstrap-result.json'))['productionApiKey'])") +VALIDATE_URL=$(python3 -c "import json;print(json.load(open('service-urls.json'))['validateclient'])") +# The payload needs a full `user` OBJECT (top-level userId alone returns an app-level error): +curl -s -X POST "$VALIDATE_URL" -H 'Content-Type: application/json' -H "Origin: https://" \ + -d "{\"data\":{\"apiKey\":\"$PROD_KEY\",\"user\":{\"userId\":\"smoke-user\",\"userSnippylyId\":\"smoke-user\",\"email\":\"smoke@example.com\"}}}" +# → HTTP 200 with a Firebase custom token whose claims carry the provisioned firebaseConfig. +# With min_instances_override = 0 the first hit is a COLD START — retry up to ~5× with +# 15 s backoff before treating an internal error as real. +``` + +Mark `3-workspace: done`. + +--- + +## Phase 4: Console + +The backend apply already produced the two artifacts the console needs: +`velt-console-config.json` (runtime config: the project's real `firebaseConfig`, every +service URL in `functionUrls`, the `sendLoginLink`/`aiChat` endpoints) and +`console-firebase.json` (Hosting config with the SPA rewrite). + +### 4.1 Fetch + verify the console bundle (pins from the manifest) + +```bash +BUNDLE_REF=$(python3 -c "import json;print(json.load(open('manifest.json'))['console']['bundleRef'])") +BUNDLE_SHA=$(python3 -c "import json;print(json.load(open('manifest.json'))['console']['sha256'])") +oras pull -o . "$BUNDLE_REF" # writes console-dist-.tar.gz +BUNDLE_TAR=$(ls console-dist-*.tar.gz) +echo "$BUNDLE_SHA $BUNDLE_TAR" | shasum -a 256 -c - # MUST print OK — hard-fail otherwise +mkdir -p console-bundle && tar -xzf "$BUNDLE_TAR" -C console-bundle +``` + +### 4.2 Stage + deploy to the host the operator chose (`CONSOLE_HOST`, Phase 0.2) + +Stage first, identical for every host: + +```bash +DEPLOY_DIR=$(mktemp -d) +cp -R console-bundle "$DEPLOY_DIR/dist" # the folder containing index.html +cp "$MODULE_DIR/terraform/velt-console-config.json" "$DEPLOY_DIR/dist/velt-console-config.json" + +# CRITICAL check before deploying: firebaseConfig.authDomain in the runtime config must +# be the domain the console is SERVED from (the CONSOLE_BASE hostname), NOT +# .firebaseapp.com. Cross-origin auth handlers break sign-in silently under +# modern browser storage partitioning. Patch the JSON if needed. +python3 -c "import json;print(json.load(open('$DEPLOY_DIR/dist/velt-console-config.json'))['firebaseConfig'].get('authDomain'))" +``` + +Non-negotiable serving rules for ANY console host (each one, if violated, breaks +the console at runtime): + +| Rule | Why | +|---|---| +| SPA fallback: every unknown path serves `dist/index.html` (200, not 301/404) | the console is a client-routed SPA; deep links like `/dashboard/config` must load the app | +| `velt-console-config.json` served at the root with `Cache-Control: no-cache` | runtime config; config changes must take effect without a bundle redeploy | +| HTTPS on the exact `CONSOLE_BASE` origin | it is baked into CORS admission, magic-link URLs, OAuth origins, and Identity Platform authorized domains | +| `Content-Type: text/javascript` on `.js`, correct types on other assets | browsers refuse module scripts with wrong MIME | +| `/__/auth/*` and `/__/firebase/*` reverse-proxied to `https://.firebaseapp.com` (Firebase Hosting hosts: automatic, no action) | the `authDomain` patch above makes the Firebase Auth SDK load its sign-in helper iframe/handler from the console's OWN domain; any host that isn't Firebase Hosting serves 404s there and Google sign-in dies silently | + +Then deploy per the operator's choice, **this was their decision; do not silently +substitute another host**: + +- **Operator's existing static host / GCP-native stack** (e.g. GCS bucket + external + HTTPS LB + Cloud CDN, or Cloud Run): upload `$DEPLOY_DIR/dist/` with their tooling + (or provision the stack in this project if they asked you to), honoring the serving + rules above. On a GCS+LB setup the SPA fallback is the one that needs care: set the + backend bucket's `not_found_page` (`errorDocument`) to `index.html`, and note GCS + serves it with a 404 status, which browsers render fine but naive health checks + flag; probe `/` (200) rather than a deep link. A Google-managed cert on the LB + needs the operator's DNS record and can take 15–60 min to go ACTIVE after DNS + resolves. Start it early and continue Phase 5 in parallel. (If Phase 5 also + provisions a GCP-native host, share ONE LB across both surfaces: one IP, one + managed cert covering both hostnames, and a host-routed URL map with a backend + bucket per surface, this halves cert wait and IP/DNS churn.) + + **The `/__/auth/*` proxy on a GCS+LB console host** (the serving rule above , + GCS buckets can't serve these paths themselves): + +```bash +# Internet NEG pointing at the project's firebaseapp.com origin, with a Host rewrite: +gcloud compute network-endpoint-groups create firebase-auth-neg --global \ + --network-endpoint-type=internet-fqdn-port --default-port=443 --project="$PROJECT_ID" +gcloud compute network-endpoint-groups update firebase-auth-neg --global \ + --add-endpoint="fqdn=${PROJECT_ID}.firebaseapp.com,port=443" --project="$PROJECT_ID" +gcloud compute backend-services create firebase-auth-backend --global \ + --load-balancing-scheme=EXTERNAL --protocol=HTTPS \ + --custom-request-header="Host: ${PROJECT_ID}.firebaseapp.com" --project="$PROJECT_ID" +gcloud compute backend-services add-backend firebase-auth-backend --global \ + --network-endpoint-group=firebase-auth-neg --global-network-endpoint-group --project="$PROJECT_ID" +# Then add path rules on the console host in the URL map (export → edit → import): +# pathRules: /__/auth/* and /__/firebase/* → firebase-auth-backend. +# ⚠ If Cloud CDN fronted the console BEFORE the rule existed, it has CACHED the 404s — +# invalidate after importing: gcloud compute url-maps invalidate-cdn-cache --path "/__/*" +# (a cache-busted curl, e.g. ?cb=$(date +%s), proves the rule works while stale +# entries are still being purged). +``` +- **Firebase Hosting site in this project** (only if the operator chose it) , + `CONSOLE_BASE` becomes `https://.web.app`; the Terraform-emitted + `console-firebase.json` already carries the SPA rewrite: + +```bash +cp "$MODULE_DIR/terraform/console-firebase.json" "$DEPLOY_DIR/firebase.json" + +# Ensure the hosting site exists first. The REST form takes siteId as a QUERY PARAM +# (a JSON-body siteId is rejected). Refresh the token — Phase 1's is hours stale by now: +TOKEN=$(gcloud auth print-access-token) +AUTH=(-H "Authorization: Bearer $TOKEN" -H "X-Goog-User-Project: $PROJECT_ID" -H "Content-Type: application/json") +curl -s -X POST "${AUTH[@]}" \ + "https://firebasehosting.googleapis.com/v1beta1/projects/$PROJECT_ID/sites?siteId=" -d '{}' + +cd "$DEPLOY_DIR" && firebase deploy --only hosting --project "$PROJECT_ID" +``` + +If the firebase CLI rejects its stored token and interactive `firebase login --reauth` +isn't possible, deploy non-interactively with a short-lived service account instead: +mint an SA with `roles/firebasehosting.admin`, `export GOOGLE_APPLICATION_CREDENTIALS=`, +and run `npx --yes firebase-tools deploy --only hosting --project "$PROJECT_ID" --non-interactive` +(delete the SA key immediately after, same hygiene as Phase 3). + +Also give `velt-console-config.json` a `no-cache` header if editing `firebase.json` +(config changes then take effect without redeploying the bundle). + +**Custom (non-`web.app`) console domains, one extra step:** Terraform already +authorizes the `velt_portal_url` hostname on Identity Platform +(`console_manage_identity_platform = true` derives `authorized_domains` from it), but +verify after apply: Identity Platform → Settings → Authorized domains must list the +`CONSOLE_BASE` hostname, or Google sign-in will reject the origin. + +### 4.3 Google sign-in (human moment #2: cannot be automated) + +You should have issued this ask back in Phase 0 (all its values are known up front) , +if so, just collect the id/secret now. If not, issue it verbatim: + +> **Operator ask (verbatim):** "Open +> `https://console.cloud.google.com/apis/credentials?project=` → Create +> Credentials → OAuth client ID → Web application. Authorized JavaScript origins: +> ``. Authorized redirect URIs: +> `https://.firebaseapp.com/__/auth/handler` AND +> `/__/auth/handler`. Save, then paste me the client +> id and secret." + +Feed them to Terraform and apply (enables the `google.com` provider on Identity +Platform): + +```bash +# add to velt.auto.tfvars: +# console_google_oauth_client_id = "" +# console_google_oauth_client_secret = "" +terraform apply +``` + +(Email-link sign-in is already enabled by `console_manage_identity_platform = true`, +but the magic-link email sends via Customer.io, without a real `CUSTOMER_IO_API_KEY` +secret version it fails at send time. Google OAuth is the primary path.) + +**Verify (phase gate):** + +```bash +HOST="" # e.g. https://.web.app for firebase-hosting +curl -sf "$HOST/velt-console-config.json" | python3 -m json.tool > /dev/null && echo "config OK" +curl -sf "$HOST/dashboard/config" | grep -q "app-root" && echo "SPA rewrite OK" +# CORS: the backend must admit the console origin +curl -s -o /dev/null -w "%{http_code}\n" -X OPTIONS \ + "$(python3 -c "import json;print(json.load(open('service-urls.json'))['consolehandler'])")" \ + -H "Origin: $HOST" -H "Access-Control-Request-Method: POST" # → 204 +``` + +Mark `4-console: done`. (The sign-in itself is proven in Phase 6.) + +--- + +## Phase 5: SDK on the customer's CDN + +May run in parallel with Phase 4. Choose the SDK version: the manifest's +`sdk.testedVersion` unless the operator pins another one, anything `>= sdk.minVersion` +is supported. **The version must equal what the app's installed wrapper targets** +(`npm ls @veltdev/client @veltdev/react` in the app). + +The release registry carries the testedVersion's SDK files as `sdk.bundleRef` (the +pristine npm tarball, sha256-pinned by the signed manifest), prefer it: + +```bash +VELT_VERSION=$(python3 -c "import json;print(json.load(open('manifest.json'))['sdk']['testedVersion'])") +SDK_REF=$(python3 -c "import json;print(json.load(open('manifest.json'))['sdk']['bundleRef'])") +SDK_SHA=$(python3 -c "import json;print(json.load(open('manifest.json'))['sdk']['sha256'])") +WORKDIR=$(mktemp -d) +oras pull -o "$WORKDIR" "$SDK_REF" # writes veltdev-sdk-.tgz +cd "$WORKDIR" +SDK_TGZ=$(ls veltdev-sdk-*.tgz) +echo "$SDK_SHA $SDK_TGZ" | shasum -a 256 -c - # MUST print OK — hard-fail otherwise +tar -xzf "$SDK_TGZ" +mkdir -p "staging/lib/sdk@$VELT_VERSION" +cp -R package/. "staging/lib/sdk@$VELT_VERSION/" +``` + +If the operator pinned a version OTHER than `testedVersion`, the registry has no +artifact for it, fetch that version from npm instead (same staging steps): +`npm pack "@veltdev/sdk@$VELT_VERSION" && tar -xzf veltdev-sdk-*.tgz`. + +Deploy `staging/` to the host the operator chose in `SDK_HOST` (Phase 0.2, **this was +their decision; do not silently substitute another host**): + +- **Operator's existing CDN / static host** (the common case): upload `staging/` + preserving the `lib/sdk@/` layout, using their normal upload tooling + (`aws s3 sync`, `rsync`, `wrangler`, CI job, …). If you don't have their upload + credentials, hand them the `staging/` directory plus the serving-rules table below + as a checklist and wait for the URL. +- **GCS + external HTTPS LB + Cloud CDN in this project**: recipe below. Start it as EARLY as Phase 5 allows: the managed cert is the + long pole (15–60 min after the DNS record is visible), and the DNS record is an + operator action (hand them the IP the moment it's reserved; feasibility was + confirmed in Phase 0.2). + +```bash +SDK_DOMAIN= +BUCKET="$PROJECT_ID-velt-sdk" + +gcloud storage buckets create "gs://$BUCKET" --project="$PROJECT_ID" --location="$REGION" +gcloud storage buckets add-iam-policy-binding "gs://$BUCKET" \ + --member=allUsers --role=roles/storage.objectViewer +gcloud storage cp -r staging/* "gs://$BUCKET/" +# Set the serving metadata on every .js (content-type + cache-control): +gcloud storage objects update "gs://$BUCKET/lib/**.js" \ + --content-type="text/javascript" \ + --cache-control="public, max-age=31536000, immutable" + +gcloud compute addresses create velt-sdk-ip --global --project="$PROJECT_ID" +gcloud compute addresses describe velt-sdk-ip --global --project="$PROJECT_ID" \ + --format="value(address)" # → HAND THIS TO THE OPERATOR NOW: A record for $SDK_DOMAIN + +# ⚠ cache-mode MUST be USE_ORIGIN_HEADERS: the default (CACHE_ALL_STATIC) silently +# REWRITES the client-facing Cache-Control to max-age=3600, overriding the object +# metadata you just set. The custom response header supplies ACAO +# on every object without per-object CORS config. +gcloud compute backend-buckets create velt-sdk-backend --gcs-bucket-name="$BUCKET" \ + --enable-cdn --cache-mode=USE_ORIGIN_HEADERS \ + --custom-response-header='Access-Control-Allow-Origin: *' --project="$PROJECT_ID" +gcloud compute url-maps create velt-sdk-lb --default-backend-bucket=velt-sdk-backend --project="$PROJECT_ID" +gcloud compute ssl-certificates create velt-sdk-cert --domains="$SDK_DOMAIN" --global --project="$PROJECT_ID" +gcloud compute target-https-proxies create velt-sdk-proxy \ + --url-map=velt-sdk-lb --ssl-certificates=velt-sdk-cert --project="$PROJECT_ID" +gcloud compute forwarding-rules create velt-sdk-fr --global --address=velt-sdk-ip \ + --target-https-proxy=velt-sdk-proxy --ports=443 --project="$PROJECT_ID" + +# Poll the cert until ACTIVE (only starts progressing once the DNS record is visible; +# FAILED_NOT_VISIBLE in the meantime is harmless — it keeps retrying on its own): +gcloud compute ssl-certificates describe velt-sdk-cert --global --project="$PROJECT_ID" \ + --format="value(managed.status)" +# After ACTIVE, first requests may still throw TLS errors for a few minutes while the +# cert propagates to the edge — retry with backoff before diagnosing. +``` + +- **Firebase Hosting site in this project** (only if the operator chose it): create a + site and deploy, site id must be globally unique; `CDN_BASE` becomes + `https://.web.app`: + +```bash +TOKEN=$(gcloud auth print-access-token) +AUTH=(-H "Authorization: Bearer $TOKEN" -H "X-Goog-User-Project: $PROJECT_ID" -H "Content-Type: application/json") +curl -s -X POST "${AUTH[@]}" \ + "https://firebasehosting.googleapis.com/v1beta1/projects/$PROJECT_ID/sites?siteId=" -d '{}' +cat > "$WORKDIR/firebase.json" << 'EOF' +{ "hosting": { "site": "", "public": "staging", + "headers": [ { "source": "**/*.js", "headers": [ + { "key": "Access-Control-Allow-Origin", "value": "*" }, + { "key": "Cache-Control", "value": "public, max-age=31536000, immutable" } ] } ] } } +EOF +cd "$WORKDIR" && firebase deploy --only hosting --project "$PROJECT_ID" +``` + +Non-negotiable serving rules for ANY host (each one, if violated, +breaks the SDK at runtime): + +| Rule | Why | +|---|---| +| Path is exactly `${CDN_BASE}/lib/sdk@/velt.js`, all files flat in that dir, `@` literal | wrapper hard-codes the path shape; chunks resolve relative to `velt.js` | +| `Access-Control-Allow-Origin` on every `.js` (app origin or `*`) | ES-module fetches are CORS-mode; this is the #1 failure mode | +| `Content-Type: text/javascript` on `.js` | browsers refuse module scripts with wrong MIME | +| HTTPS + no auth wall | browser fetches anonymously | +| `Cache-Control: public, max-age=31536000, immutable` (recommended) | version-pinned path = cache-forever safe. ⚠ Verify the header **as served**, not just at the origin, because CDN layers can rewrite it (Cloud CDN's default cache mode caps it at 3600; see the recipe above) | + +### 5.1 Assemble the app's `selfHosted` config (deliverable: always produce this) + +Serving the SDK from the customer's CDN only moves the *code*; the SDK's **runtime +endpoints** still default to Velt SaaS. The `config.selfHosted` object repoints every +endpoint at the customer's own deployment, without it there is no zero-egress. Every +value is already on disk from earlier phases; generate it, don't hand-type it: + +```bash +cd
# where manifest.json / service-urls.json / the state file live +# Fresh shell? Re-export the Phase 2.1 module dir first: +# export MODULE_DIR="$(pwd)/velt-backend-module" +terraform -chdir="$MODULE_DIR/terraform" output -json enabled_modules > enabled-modules.json +python3 - << 'PYEOF' +import json +urls = json.load(open('service-urls.json')) +modules = json.load(open('enabled-modules.json')) +# firebaseConfig: the Terraform-EMITTED file inside $MODULE_DIR (NOT the +# console-deployed copy, whose authDomain was patched to the console domain in Phase 4). +import os +fb = json.load(open(os.path.join(os.environ.get('MODULE_DIR', 'velt-backend-module'), + 'terraform', 'velt-console-config.json')))['firebaseConfig'] +project = fb['projectId'] + +cfg = { + # Fail-closed: any endpoint not explicitly injected resolves to an inert + # velt://self-hosted-disabled/* sentinel (zero egress) instead of Velt SaaS. + 'strict': True, + # MUST be the resolved closure from Terraform (CLI output) — never hand-derive it; + # endpoints of modules missing from this list degrade to inert sentinels by design. + 'deploymentProfile': modules, + 'firebaseConfig': fb, + 'firebaseNotificationsDatabaseURL': f'https://{project}-notifications.firebaseio.com', + 'cloudFunction': { + # The strict-core set — all served by `core`-module services, always provisioned: + 'validateClient': urls['validateclient'], + 'sdkProxy': urls['foldersapibe'] + '/v2/core/a', + 'setEncryptedData': urls['setencrypteddata'], # single-region collapse of the x5 regional split + 'getNotificationsForDocuments': urls['getnotificationsfordocuments'], + 'getPlanDetails': urls['getplandetails'], + 'getAllowedDocuments': urls['getalloweddocuments'], + 'sa': urls['sa'], + }, +} +# Module-gated endpoints — add ONLY when the service was provisioned (present in +# service_urls). Undeployed ones need no entry: strict mode already inert-sentinels +# them, and their modules are absent from deploymentProfile anyway. +for cf_key, svc in [('chatgptCompletion', 'chatgptcompletion'), # ai + ('getIceServers', 'geticeservers'), # huddle-webrtc + ('whisperTranscription', 'whispertranscription')]: # recorder-media + if svc in urls: + cfg['cloudFunction'][cf_key] = urls[svc] +# recorder-media's remaining endpoints (convertRecording / processRecording / +# videoBackend / screenshot) sit behind SaaS URL-map paths whose self-host mapping +# ships with the recorder deployment notes — consult Velt release notes if the +# deployment includes recorder-media rather than guessing service names. +# If tfvars set data_regions, mirror it VERBATIM as cfg['dataRegions'] ([] and null are +# DIFFERENT fleets — never coalesce). If data_regions was never set, omit the key. + +json.dump(cfg, open('velt-selfhosted-config.json', 'w'), indent=2) +print('wrote velt-selfhosted-config.json') +PYEOF +``` + +Record the path as `artifacts.selfHostedConfigPath` in the state file. Point the app at +it. `proxyDomain` is origin-only, and **always pin `version`**: + +```tsx +import selfHosted from './velt-selfhosted-config.json'; + +", version: "", selfHosted }}> +``` + +(Vue/vanilla: `initVelt(apiKey, { proxyDomain, version, selfHosted })`.) If the app has +a CSP, add `CDN_BASE` to `script-src` and remove `cdn.velt.dev`. Without `selfHosted` +the SDK code loads from the CDN but every API call still goes to Velt SaaS; the Phase 6 +zero-egress audit would fail. + +**Verify (phase gate):** + +```bash +# Fresh temp path — a fixed /tmp filename survives failed curls from earlier runs and +# a stale file will pass `head` with the WRONG version banner: +CHECK_JS=$(mktemp) +curl -sS -D - -o "$CHECK_JS" "$CDN_BASE/lib/sdk@$VELT_VERSION/velt.js" -H "Origin: https://" | head -20 +head -c 60 "$CHECK_JS" # starts with: var SNIPPYLY_VERSION = ''; +curl -s -o /dev/null -w "%{http_code}\n" "$CDN_BASE/lib/sdk@$VELT_VERSION/velt-comment.js" # 200 — chunks uploaded too +``` + +Mark `5-sdk-cdn: done`. + +--- + +## Phase 6: End-to-end acceptance (the payoff) + +Human moment #4: the operator (or you, with browser tooling) proves the full loop. + +**First, ASK the operator: "Do you have a test app (or a dev build of your real app) +we can wire the SDK into?"** Do not silently spin up a synthetic page. + +- **They have one (preferred):** apply the Phase 5 changes there, swap the SDK + source to `proxyDomain` + pinned `version`, add the generated `selfHosted` object + and the production key, and run the checks below in THAT app. A real app also + exercises what a synthetic page can't: CSP headers, the bundler/framework wrapper + (`@veltdev/react` etc.), and the app's own auth flow. +- **They don't:** create a minimal test page yourself (plain HTML, SDK loaded from + `CDN_BASE`, the same Phase 5 config, an identified test user + test document, the + comments + comment-tool components. Follow the SDK quickstart for the current + init shape) and serve it on a localhost port. Tell the operator it's a stand-in + and that the loop should be re-proven inside their app when one exists. + +Diagnostic rule either way: if comments misbehave, read the SDK's `sa` error +telemetry (Cloud Run logs of `velt-sa`, which carries Firestore listener errors with +index-creation links) BEFORE probing backend endpoints by hand. Hand-built curl +calls to callable endpoints carry no Firebase ID token, so auth-gated endpoints +return errors like "Api key not found" that look like product bugs but aren't +. + +1. **Console sign-in:** open ``, sign in with Google + as `OWNER_EMAIL` → lands on the dashboard showing the bootstrapped workspace and + both API keys. (Admins are matched by email; the first sign-in with a seeded email + lands in the workspace.) +2. **SDK loop:** open the app chosen above (with the Phase 5 config, `proxyDomain` + + `version` + the generated `selfHosted` object, and the production key), + DevTools → Network, filter `velt`: + - `velt.js` + chunks load from `CDN_BASE`, **nothing from `cdn.velt.dev`**; + - `window.Velt.version` === the pinned version; + - create a comment → it renders; the API calls in the Network tab go to the + customer's own `*.run.app` services. +3. **Cross-surface proof:** the comment created in step 2 is visible in the console's + data browser (comments/documents view), SDK → customer backend → customer console, + one loop, zero Velt infrastructure. +4. **Zero-egress audit:** across both browser sessions, the network log contains **no + request to any `velt.dev` or Velt-owned host** (no `cdn.velt.dev`, + `console.velt.dev`, `api.velt.dev`, Velt Sentry/analytics). + +**Verify (final gate):** all four pass. Mark `6-acceptance: done`. + +### Final report: deliver this to the operator + +- Console URL, and where `velt-console-config.json` lives. +- Workspace id + where the bootstrap secrets are stored (testing key, production key, + REST header pair), plus the reminder that the production key's `allowedDomains` is + the SDK gate (managed in the console thereafter). +- SDK CDN base + pinned version, and the app config snippet in use, **including the + full generated `selfHosted` config object** (`velt-selfhosted-config.json` contents + inline in the report; it contains no secrets: only public URLs, the web-app + firebaseConfig, and the module list). This is the artifact the customer's app team + actually needs to integrate. +- Deployed self-host release version (from the manifest), profile + opt-in modules, + and the tfvars/state-file locations. +- Anything skipped or degraded (e.g. email-link sign-in pending a Customer.io key; + optional modules not enabled), and any FAILED_PRECONDITION index warnings seen in + logs (report these to Velt, Velt uses them to keep indexes complete in future releases). + +### Upgrades (summary) + +A new self-host release = a new manifest. Diff the new manifest against +`release` in the state file; re-run only the phases whose pins changed +(backend → Phase 1.4 image copy + Phase 2 applies with the new module archive; +console → Phase 4; SDK → Phase 5 into a NEW `lib/sdk@/` folder, keeping the +old one for instant rollback). The full agent-executable procedure (delta +computation, compatibility fences for `sdk.minVersion` and console config schema, +secrets-delta seeding, Terraform state carry-over, and per-component rollback +anchors) is the companion [Upgrade guide](/self-hosting/full/gcp/upgrade). + +--- + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| Any firebase*/identitytoolkit REST call → 403 `SERVICE_DISABLED` mentioning a quota project | Missing header | Add `X-Goog-User-Project: $PROJECT_ID` | +| `gcloud artifacts docker images scan` hangs indefinitely with no output | First run wants to install the `local-extract` gcloud component and is waiting on a hidden interactive Y/n prompt | Always pass `--quiet` (or pre-install: `gcloud components install local-extract --quiet`) | +| Image scan reports CRITICAL/HIGH findings | Expected for a full Node runtime image; most are unfixed Debian base-OS CVEs | Compare against the signed manifest's `backend.knownFindings` (Phase 1.5 decision rule); only NOT-listed findings are a stop-and-contact-Velt condition | +| `gcloud` works but Terraform 401s | ADC ≠ CLI token | `gcloud auth application-default login` (or the 60-min `GOOGLE_OAUTH_ACCESS_TOKEN` fallback) | +| `firebase` CLI "credentials no longer valid" | Separate token store from gcloud | `firebase login --reauth` (or use the REST equivalents shown in Phase 1) | +| API enable fails `UREQ_PROJECT_BILLING_NOT_FOUND` | Billing not linked | Human moment #1 | +| RTDB instance create → `Blaze plan required` | Billing not linked | Human moment #1 | +| `addFirebase` fails on a fresh project | `firebase.googleapis.com` not yet enabled, or the project-ID form flaking | Enable APIs first (Phase 1.2 order); retry with the project NUMBER | +| A `curl` to a `…/$VAR:someMethod` REST URL returns an HTML 404 | zsh parameter modifier: `$VAR:a…` is parsed as the `:a` (absolute path) modifier and the URL is silently mangled | Brace every variable followed by a colon: `${VAR}:someMethod` | +| `defaultBucket` POST fails (404 OR 400 INVALID_ARGUMENT) on every variant | Fresh-project quirk | App Engine fallback in Phase 1.3 (`gcloud app create`), then link the bucket with `buckets/:addFirebase` (also in 1.3); GET on defaultBucket 404s until linked | +| `docker pull ` → "no matching manifest for linux/arm64" | Image is linux/amd64-only; bare pull fails on Apple Silicon (may falsely succeed if a stale local cache has the layers) | `docker pull --platform linux/amd64`, or better `crane copy` (Phase 1.4) | +| Managed cert stuck `FAILED_NOT_VISIBLE` | DNS record not created yet / not propagated; harmless, the cert keeps retrying on its own | Confirm the A record exists and resolves; cert goes ACTIVE 15–60 min after visibility. No LB-side action needed | +| SDK hostname must change mid-run (zone turned out untouchable) | Phase 0.2 DNS feasibility check skipped or answer changed | Create a NEW managed cert for the new hostname, swap it onto the existing HTTPS proxy (`target-https-proxies update --ssl-certificates=`), delete the old cert, update `CDN_BASE` in the state file; LB/bucket/IP all survive unchanged | +| Cert just went ACTIVE but curl throws TLS/SSL errors | Edge propagation lag after activation | Retry with backoff for a few minutes before diagnosing | +| Served `Cache-Control` ≠ the object metadata you set (e.g. `max-age=3600`) | Cloud CDN backend-bucket default cache mode `CACHE_ALL_STATIC` rewrites the client-facing header | `gcloud compute backend-buckets update --cache-mode=USE_ORIGIN_HEADERS` | +| provision-cli exits instantly, log mentions `FUNCTION_TARGET` | Image's default entrypoint is the functions-framework launcher | `--entrypoint node` on the `docker run` (Phase 3) | +| provision-cli: SA key file empty/missing despite correct `-v` flag | Host path (e.g. `/tmp`) not shared into the Docker VM (macOS Docker Desktop / colima) | Mount from a shared dir, e.g. `$HOME/velt-provision-keys` | +| `bootstrap-result.json` fails to parse | Stray log lines leaked into stdout around the result JSON | Extract the last top-level JSON object (brace-match from the final `"schemaVersion": 1`) | +| First service calls return internal errors, then succeed | Cold start with `min_instances_override = 0` | Retry ~5× with 15 s backoff before diagnosing | +| Hosting site create REST → error about siteId | siteId passed in the JSON body | Pass it as a query param: `…/sites?siteId=` | +| First Eventarc trigger: "Permission denied … Eventarc Service Agent" | Async agent-permission propagation after API enable | Wait ~5 min, re-apply (retryable by design) | +| Cloud Tasks queue create → `FAILED_PRECONDITION: … existed too recently` | Reused project + same region within ~7 days of a `terraform destroy` (queue-name tombstone) | Phase 0 reused-project gate: fresh project, or wait out the tombstone in the SAME region; do NOT switch regions (see next row) | +| Cloud Run services fail `Project failed to initialize in this region due to quota exceeded` + `Resource readiness deadline exceeded` on every service in a NEW region | Regions-per-project cap (default 5): 4 pinned `setencrypteddata` regions + the original home region already fill it, and destroy does not release region initialization | Go back to the original region (or a fresh project); a genuinely new region needs a Cloud Run quota increase first | +| Cloud Run revision rejected on memory quota | Fresh-project `MemAllocPerProjectRegion` = 400 GiB | The module caps instances to fit (`region_memory_quota_gib`); raise the variable only if your quota was raised | +| Apply never converges, perpetual diffs every plan | Provider/API echo drift | Should not happen on the pinned module version; capture `terraform plan` output and report to Velt | +| `terraform apply` fails creating Identity Platform config | Singleton already exists (Phase 1 initializeAuth) | The `terraform import` step in Phase 2.3 | +| `terraform import` of Identity Platform config fails "not found" | Phase 1 initializeAuth silently no-oped (e.g. missing Content-Type on the POST); the singleton does NOT exist | Skip the import and let the module create it; the Phase 1.3 verify probe (admin/v2 config GET) tells you which case you are in | +| `terraform apply` 403s on firebaserules with a quota-project error | `USER_PROJECT_OVERRIDE` / `GOOGLE_BILLING_PROJECT` not exported (they were previously only shown inside the import snippet) | `export USER_PROJECT_OVERRIDE=true GOOGLE_BILLING_PROJECT="$PROJECT_ID"` before any apply (Phase 2.3) | +| Service revision fails on a missing secret version | A `secrets_to_seed` id wasn't seeded | Seed it (`gcloud secrets versions add`), re-apply | +| Calls fail `CACHEDDATA_URL_NOT_CONFIGURED` | Between the two cacheddata applies | Finish Phase 2.5 (fail-closed by design, never a Velt URL) | +| provision-cli: SA key "not found" inside container | Key mounted under `/tmp` | Mount under `/keys` | +| provision-cli silent for many minutes after result computed | In-process composite-index creation | Normal; wait for exit 0 (often tens of minutes; can exceed two hours depending on Firestore load) | +| Production DB has fewer composite indexes than sdktest (partial set despite exit 0) | Index converge incomplete on the production store DB | Phase 3 gate: compare counts vs sdktest; re-run via `documentmigrationhandler` `{"eventType":"create_firestore_indexes","dbId":""}` until 200 | +| Google sign-in on a non-Firebase-Hosting console: popup opens then nothing / handler 404 | `/__/auth/*` + `/__/firebase/*` not served on the console domain (authDomain = own domain requires them) | Phase 4.2 proxy recipe (internet NEG + Host rewrite to `.firebaseapp.com`); if CDN-fronted, invalidate cached 404s after adding the path rules | +| Comments render then vanish / never persist; SDK `sa` telemetry shows Firestore listener errors "The query requires an index" (`allMultiThreads`, `allDocumentUsers`, `allOrganizationUsers`) | Partial composite-index set on the production DB (see the Phase 3 gate; the CLI's prod-key index pass can silently end early) | Compare index counts vs sdktest; re-run via `documentmigrationhandler` `{"eventType":"create_firestore_indexes","dbId":""}` until 200, then re-test | +| SDK logs "Documents provided are all denied"; every document probe PERMISSION_DENIED despite valid auth | `apiKey/` doc missing from the production store DB; security rules `get()` this doc, and a missing doc fails the rule outright (a missing field would fall back to `public`) | Phase 3 gate: `curl` the doc; if 404, PATCH it with `metadata.defaultDocumentAccessType = "public"` (recipe in the gate), then re-test | +| Manual probe of `getalloweddocuments` returns "Api key not found" despite `apiKey` in the body | The endpoint authenticates via the Firebase ID token ONLY (`request.auth.token` claims); body `apiKey` is ignored by design. The SDK itself never calls this endpoint | Not a product failure, so don't chase it. Diagnose comment issues via the SDK's `sa` error telemetry (Firestore listener errors) instead | +| Console boots but sign-in fails oddly / placeholder project | `velt-console-config.json` missing at web root or invalid (app fails soft onto build defaults, one `[self-host-config]` console error) | `curl $HOST/velt-console-config.json`; must be valid JSON with apiKey/projectId/appId/databaseURL | +| Console CORS error on `cloudfunctions.net/` | Function missing from `functionUrls` → name-based dispatch fallback | Redeploy the Terraform-emitted config; the emitted file lists every provisioned service | +| Google login "succeeds" then returns to the sign-in page | `authDomain` is `.firebaseapp.com` (cross-origin credential handoff dropped by browser partitioning) | Set `authDomain` to the console's own domain (Phase 4.2 check) | +| Google `Error 400: redirect_uri_mismatch` | Console-domain redirect URI not on the OAuth client | Human moment #2: add `https:///__/auth/handler` | +| `auth/unauthorized-domain` | Console origin not in Auth authorized domains | `console_manage_identity_platform` manages these; re-apply, or add the custom domain | +| SDK: CORS error on `velt.js` | Missing ACAO header on the CDN | Add to ALL files, not just `velt.js` | +| SDK: "Failed to load module script… MIME" | `.js` served as `text/plain` | Set `Content-Type: text/javascript` | +| SDK: 404 on `/lib/sdk@/velt.js` | Path layout wrong, `@` mangled by upload tool, or pinned version ≠ uploaded folder | Curl the exact URL; fix layout; pin == folder | +| SDK loads, one feature silently missing | Not all package files uploaded | Re-upload the complete `package/` contents | +| Backend logs show `FAILED_PRECONDITION` needing a composite index | Missing composite index on the default DB | Add the index to `default_db_composite_indexes` in tfvars, apply; report to Velt | +| Works, then breaks after an app npm upgrade | Wrapper bumped but CDN folder/version pin not updated | Follow the Upgrade guide: new `lib/sdk@/` folder first, then repoint `version` | + +--- + +*This guide is evergreen: every version-specific value (image digest, module sha, +console/SDK versions, registry) comes from the signed release manifest at run time, +so the guide text itself does not change per release. The canonical copy lives on the +Velt docs site. (`guideVersion` in the state file tags the state-file contract for +resume, not a release.) Release enumeration: +`oras pull -o . us-docker.pkg.dev/velt-sdk/velt-releases/velt-selfhost-index:latest` +(writes `index.json`, newest first).* diff --git a/self-hosting/full/gcp/overview.mdx b/self-hosting/full/gcp/overview.mdx new file mode 100644 index 000000000..44a26274c --- /dev/null +++ b/self-hosting/full/gcp/overview.mdx @@ -0,0 +1,179 @@ +--- +title: "Get Started on GCP" +sidebarTitle: "Get Started" +description: "Deploy Velt full self-hosting into your own GCP project. Hand the install guide to an AI coding agent, answer a handful of inputs, and complete the five human steps." +--- + +You do not run this deployment by hand. You give an AI coding agent the [Install guide](/self-hosting/full/gcp/install), and it executes the phases, stopping to ask you whenever a step needs a human. + +Budget roughly 2 to 4 hours of wall clock. Most of that is waiting on Terraform and Firestore index creation, not typing. + + + New to full self-hosting? Read the [Overview](/self-hosting/full/overview) first for what the deployment contains and how it differs from [partial self-hosting](/self-hosting/partial/overview). + + + + + + +**People and access** + +- A GCP project you can administer (or create), with billing enabled. +- Ability to create a Google OAuth client for console sign-in. This is one human console step. +- Someone available to run an AI coding agent, or the shell commands themselves, for the duration. + +**Tooling** + +| Tool | Why | +| --- | --- | +| `gcloud`, `firebase` CLI | Project bootstrap, Hosting, Auth | +| Terraform ≥ 1.5 | Backend apply | +| `oras` | Pull manifest, module, console, and SDK artifacts | +| `cosign` | Verify signatures | +| Node ≥ 20 | Deployment-profiles CLI inside the module archive | +| `curl`, `python3`, `jq` | Verify steps | + +The agent checks exact versions itself in Phase 0 and stops if anything is missing. + + + + + +Decide these once, up front. The agent asks for all of them at the start of Phase 0 rather than interrupting you later. + +| Input | Meaning | +| --- | --- | +| `PROJECT_ID` / `REGION` | Where Cloud Run and Firebase live (`us-central1` recommended) | +| `PROFILE` + `OPT_IN_MODULES` | Feature surface, see [Deployment profiles](/self-hosting/full/gcp/reference#deployment-profiles) | +| `OWNER_EMAIL` / admin emails | Console login identities | +| `appDomains` | Production API key `allowedDomains` (SDK gate) | +| `GOOGLE_GEN_AI_API_KEY` / `ANTHROPIC_API_KEY` | Operator-owned LLM keys | +| Console host + SDK CDN base | Where static assets are served | + + + The Gemini and Anthropic keys are **required even on the `core` profile**. Memory embeddings and knowledge search run on Gemini; agent paths use Anthropic. Placeholder values pass the install and then fail at runtime, so use real keys you control. + + + + + + +This is the step that does the work. Point your AI coding agent at these two pages: + +- [Install guide](/self-hosting/full/gcp/install), the phase-by-phase procedure it executes. +- [Reference](/self-hosting/full/gcp/reference), the field-level contract it checks shapes against. + +Then start it with a prompt like this, filled in with the values you decided in step 2: + +```text +You are deploying Velt full self-hosting into our own GCP project. + +Read the Velt full self-hosting Install guide and Reference pages I linked +before you begin, and follow the Install guide phase by phase. Do not invent +your own procedure. + +Inputs: + PROJECT_ID = + REGION = us-central1 + PROFILE = core + OPT_IN_MODULES = + OWNER_EMAIL = + CONSOLE_BASE = + CDN_BASE = + +Maintain velt-selfhost-state.json after every phase so this run can be +resumed in a fresh session. Stop and ask me whenever a step needs a human. +``` + +Keep `velt-selfhost-state.json` in your working directory. It is what lets a fresh session pick up where the last one stopped. + + + + + +Five moments need a person. The agent pauses and asks at each one. + +| Moment | What you do | +| --- | --- | +| Billing | Link a billing account to the GCP project | +| OAuth client | Create the Google OAuth client for console sign-in | +| Infosec scan | Sign off the image scan against the manifest's signed findings | +| DNS | Add the record, only if you use a custom domain | +| Sign-in test | Sign in to the console for the first time | + + + + + +Serving the SDK from your CDN only moves **code**. Runtime still defaults to SaaS until you pass `selfHosted`. + +```tsx +import selfHosted from './velt-selfhosted-config.json'; + + +``` + +Vanilla and Vue use `initVelt(apiKey, { proxyDomain, version, selfHosted })`. + +**Rules that break production if violated:** + +1. Path on the CDN is exactly `/lib/sdk@/velt.js` with **all** chunks flat in that directory (`@` is literal). +2. CORS: `Access-Control-Allow-Origin` on every `.js` file (app origin or `*`). This is the number one failure mode. +3. `selfHosted.strict: true` and `deploymentProfile` set to Terraform's resolved `enabledModules`, never hand-derived. +4. CSP: allow your CDN in `script-src`, and remove `cdn.velt.dev` once you cut over. + +Field-level shape: [Reference, `selfHosted` config](/self-hosting/full/gcp/reference#sdk-selfhosted-config). + + + + + +The deployment is complete only when all four pass: + +1. Console sign-in as a seeded admin lands on the dashboard with the workspace and keys. +2. The app loads `velt.js` and its chunks from **your** CDN, and `window.Velt.version` matches the pin. +3. Creating a comment in the app persists and appears in the console data browser. +4. A Network audit on both the app and console sessions shows **no** requests to `velt.dev` or other Velt-owned hosts. + + + + + +## What the agent does + +| Phase | What happens | You care because | +| --- | --- | --- | +| **0** | Pull and cosign-verify the umbrella manifest; collect inputs; preflight tooling | A wrong or unsigned release means stop | +| **1** | Bootstrap GCP/Firebase; copy the image **by digest** into *your* Artifact Registry; infosec scan sign-off | The image never runs from Velt's registry long-term | +| **2** | Terraform apply (Cloud Run, queues, IAM, secrets, console config emission) | The profile decides which services exist | +| **3** | Provision CLI: workspace, API keys, seeded `planInfo` | First admin and keys for the app | +| **4** | Deploy console bundle and `velt-console-config.json`; OAuth; sign-in test | Admin surface on your domain | +| **5** | Host the SDK on your CDN; generate `velt-selfhosted-config.json`; wire the app | Zero-egress requires `selfHosted`, `proxyDomain`, and a pinned `version` | +| **6** | Acceptance: console login, create a comment, see it in the console data browser, Network audit | Done only when all four pass | + +Maintain `velt-selfhost-state.json` after every phase so a fresh session can resume without guessing. + +## Troubleshooting + +| Symptom | Likely cause | +| --- | --- | +| SDK never loads | Missing CORS on CDN `.js` files, or path is not `/lib/sdk@/velt.js` | +| SDK loads but data goes to Velt, or features are dead | Missing or incomplete `selfHosted`; `strict` is false; `deploymentProfile` is wrong | +| Console boots but can't sign in | OAuth client or authorized domains not pointing at the console host | +| Callable returns "Api key not found" from a hand-built curl | Expected. Callables need a Firebase ID token, so use the app or console instead | +| Comment create fails with index errors | Check `velt-sa` Cloud Run logs for index-creation links, and report gaps to Velt | + +Full tables live in the [Install guide's troubleshooting appendix](/self-hosting/full/gcp/install#troubleshooting). + +## Upgrading later + +Upgrades are a delta, not a reinstall. The [Upgrade guide](/self-hosting/full/gcp/upgrade) pulls and verifies the target manifest, diffs it against the `release` recorded in your state file, and runs only the tracks that changed. + +Releases are **migration-free** today: new backend code reads existing data shapes. If that ever changes, the release notes and the manifest call it out and ship a dedicated migration runbook. diff --git a/self-hosting/full/gcp/reference.mdx b/self-hosting/full/gcp/reference.mdx new file mode 100644 index 000000000..1f41fa9c5 --- /dev/null +++ b/self-hosting/full/gcp/reference.mdx @@ -0,0 +1,387 @@ +--- +title: "Reference" +description: "Field-level reference for Velt full self-hosting: release registry, umbrella manifest schema, trust verification, deployment profiles, the SDK selfHosted config, console runtime config, and acceptance criteria." +--- + +Field-level reference for the self-hosting release contract: registry layout, umbrella manifest, deployment profiles, runtime config shapes, and trust verification. + + + **New here?** Start with the [Overview](/self-hosting/full/overview) for concepts, then the [Get Started](/self-hosting/full/gcp/overview) page. Execute steps with the [Install](/self-hosting/full/gcp/install) and [Upgrade](/self-hosting/full/gcp/upgrade) guides. When those guides and this Reference disagree on *shapes and contracts*, **this Reference wins**. + + +## Release registry + +All customer-facing artifacts live in one public Artifact Registry repo (anonymous read): + +``` +us-docker.pkg.dev/velt-sdk/velt-releases/ +``` + +| Artifact | Tag / ref | Contents | +| --- | --- | --- | +| `velt-functions` | ``, digest | Backend container image (cosign-signed) | +| `velt-backend-module` | `` | Terraform module archive + compiled deployment-profiles CLI | +| `velt-console-dist` | `` | Console static bundle tarball | +| `velt-sdk-dist` | `` | Pristine `@veltdev/sdk` npm tarball (testedVersion) | +| `velt-selfhost-manifest` | `` (immutable), `latest` (mutable) | Umbrella manifest JSON (cosign-signed) | +| `velt-selfhost-index` | `latest` (mutable) | Enumerates published releases (newest first) | + +Pull with [`oras`](https://oras.land). Non-image artifacts are OCI artifacts; the image is a normal container. + +**Discovery vs trust:** mutable tags (`latest`) are for discovery only. Always resolve to a digest and verify the cosign signature before deploying. + +```bash +RELEASE_REGISTRY="us-docker.pkg.dev/velt-sdk/velt-releases" + +oras pull -o . "$RELEASE_REGISTRY/velt-selfhost-index:latest" # list versions +oras pull -o . "$RELEASE_REGISTRY/velt-selfhost-manifest:0.9.4" # pin a release +# or: .../velt-selfhost-manifest:latest +``` + +--- + +## Umbrella version semantics + +`selfHostVersion` (e.g. `0.9.4`) is the customer-visible release number. + +| Bump | Meaning | +| --- | --- | +| **Major** | Breaking customer deployment contract (Terraform vars, container env, console config `schemaVersion`, provision-cli flags). Read migration notes before upgrading. | +| **Minor** | New modules/functions/features; backwards-compatible `terraform apply`. | +| **Patch** | Fixes, base-image security rebuilds, or single-component bumps (unchanged components keep prior pins). | + +Component versions **do not** need to equal the umbrella version (a console-only fix can ship as self-host `1.2.1` pinning console `1.2.1` + image `1.2.0`). + +**Immutability:** a given `velt-selfhost-manifest:` is never rewritten. Fixes publish a new patch version. + +**Migration-free contract (current):** releases do not require customer-side data migrations. If that ever changes, release notes and the manifest call it out explicitly. + +--- + +## Manifest schema (`schemaVersion: 2`) + +```jsonc +{ + "schemaVersion": 2, + "selfHostVersion": "0.9.4", + "releasedAt": "2026-07-22T10:00:00.000Z", + "registry": "us-docker.pkg.dev/velt-sdk/velt-releases", + "backend": { + "image": "us-docker.pkg.dev/velt-sdk/velt-releases/velt-functions:0.9.4", + "digest": "sha256:…", + "imageByDigest": "us-docker.pkg.dev/velt-sdk/velt-releases/velt-functions@sha256:…", + "terraformRef": "container-v0.9.4", + "moduleRef": "us-docker.pkg.dev/velt-sdk/velt-releases/velt-backend-module:0.9.4", + "moduleSha256": "…", + "knownFindings": [ // optional — signed accepted scan findings + { + "id": "CVE-…", + "package": "…", + "severity": "CRITICAL", + "fixAvailable": false // publisher refuses fixAvailable: true + } + ] + }, + "console": { + "version": "…", + "bundleRef": "us-docker.pkg.dev/velt-sdk/velt-releases/velt-console-dist:…", + "sha256": "…", + "configSchemaVersion": 1 // must match Terraform-emitted console config + }, + "sdk": { + "package": "@veltdev/sdk", + "testedVersion": "6.0.0", + "minVersion": "5.8.0", + "bundleRef": "us-docker.pkg.dev/velt-sdk/velt-releases/velt-sdk-dist:6.0.0", + "sha256": "…" + }, + "releaseNotes": "https://docs.velt.dev/self-hosting/releases/v0.9.4" +} +``` + +### Field rules + +| Field | Rule | +| --- | --- | +| `schemaVersion` | Consumers must tolerate unknown additive fields. Renames/removals bump the schema (v1 to v2 replaced GCS URLs with registry refs). | +| `backend.digest` / `imageByDigest` | What Terraform and image copy must pin. Prefer digest over tag. | +| `backend.moduleSha256` | Hard-fail the install if the pulled module archive doesn't match. | +| `backend.knownFindings` | Optional. Customer scan sign-off diffs *your* Trivy/Artifact Analysis output against this signed list. | +| `console.configSchemaVersion` | Cross-repo gate: must equal what the backend Terraform emits (`velt-console-config.json`). | +| `sdk.testedVersion` | Default install pin (also available as `velt-sdk-dist`). | +| `sdk.minVersion` | Oldest SDK the backend still supports; `min ≤ tested`. Non-tested pins may use npm instead of the registry tarball. | + +### Index artifact + +```jsonc +{ + "schemaVersion": 2, + "releases": [ + { + "selfHostVersion": "0.9.4", + "releasedAt": "…", + "manifestRef": "us-docker.pkg.dev/velt-sdk/velt-releases/velt-selfhost-manifest:0.9.4" + } + ] +} +``` + +Sorted newest-first by semver. + +### Schema version 1 (legacy) + +Older manifests used GCS URLs (`moduleUrl` / `bundleUrl`) instead of registry refs. +**New installs must use `schemaVersion: 2`.** If you encounter a v1 manifest, contact Velt. + +--- + +## Trust model + +### Verify the manifest (mandatory) + +```bash +RELEASE_REGISTRY="us-docker.pkg.dev/velt-sdk/velt-releases" +TAG="0.9.4" # or latest + +MANIFEST_DIGEST=$(oras manifest fetch --descriptor \ + "$RELEASE_REGISTRY/velt-selfhost-manifest:$TAG" \ + | python3 -c "import json,sys;print(json.load(sys.stdin)['digest'])") + +cosign verify \ + --certificate-identity-regexp 'https://github\.com/[^/]+/shared-firebase-function/.*' \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com \ + "$RELEASE_REGISTRY/velt-selfhost-manifest@$MANIFEST_DIGEST" +``` + +The certificate identity is Velt's GitHub Actions OIDC signer for this release pipeline +(repo path appears in the regex by design: you do not need access to that repository). +Only deploy releases that verify successfully. + +### Trust chain + +``` +cosign(manifest@digest) + ├─ backend.digest → cosign(image@digest) independently + ├─ backend.moduleSha256 → sha256 of pulled module tarball + ├─ console.sha256 → sha256 of console-dist tarball + └─ sdk.sha256 → sha256 of sdk tarball (testedVersion) +``` + +Optional SBOM attestations on console/SDK artifacts (SPDX via cosign) are available for infosec inventory via `cosign verify-attestation`. + +### Image scan sign-off + +After copying the image into **your** registry, re-scan it. Diff CRITICAL/HIGH findings against `backend.knownFindings`. Fixable findings (`fixAvailable: true`) are never accepted in a published manifest. If you see one, stop and contact Velt. + +--- + +## Deployment profiles + +Customers pick a **curated profile**, then optionally opt in modules. There is no per-function control plane. + +### Curated profiles + +| Profile | Modules included | +| --- | --- | +| `core` | `core` | +| `core+recording` | `core`, `recorder-media` | +| `core+ai+agents` | `core`, `ai`, `agents` | +| `full` | `core`, `rest-api`, `recorder-media`, `ai`, `agents`, `huddle-webrtc` | + +### Module catalog + +| Module | Availability | Role | +| --- | --- | --- | +| `core` | always on | Auth, data plane, workspace, notifications, agents runtime, analytics/debugger backends, console backend, index tooling | +| `rest-api` | selectable | `*apibe` REST dispatchers, GDPR workers, document migration / folder-move workers | +| `recorder-media` | selectable | Recording, Whisper, screenshots, related tools | +| `ai` | selectable | AI completion / chat Cloud Functions (BYO LLM keys) | +| `agents` | selectable | Additional agents surfaces beyond the runtime already in `core` (profile `core+ai+agents`) | +| `huddle-webrtc` | selectable | ICE / huddle (`getIceServers`) | +| `integrations-workflow` | off by default | Workflow engine + connectors | +| `migrations` | off by default | Operational migration / data tooling (often useful with the full console) | + +### Opt-in + +Pass comma-separated module ids at install time (Terraform / guide `OPT_IN_MODULES`), e.g. `migrations,ai`. Only modules from the catalog above are valid: the CLI rejects unknown ids. + +### `dataRegions` (optional) + +Regional `setencrypteddata*` variants can be filtered. Omit the key for the default "all catalog regions" shape; `[]` and omitted are **different** fleets, so never coalesce them when generating SDK config. + +### Emitted profile document + +Terraform / the deployment-profiles CLI writes `velt-deployment-profile.json` (v1): + +```jsonc +{ + "schemaVersion": 1, + "profile": "core", + "optInModules": ["migrations"], + "enabledModules": ["core", "migrations"], // resolved closure — source of truth + "provisionedFunctions": ["validateclient", "…"], + "dataRegions": null // or string[] +} +``` + +Console and SDK consume **`enabledModules` verbatim**, and never re-derive the dependency closure. + +--- + +## SDK `selfHosted` config + +Generated during install (Phase 5.1) as `velt-selfhosted-config.json`. Contains **no secrets**, only public URLs, web-app `firebaseConfig`, and the module list. + +```jsonc +{ + "strict": true, + "deploymentProfile": ["core", "rest-api", "ai"], // === enabledModules from Terraform + "firebaseConfig": { + "apiKey": "…", + "authDomain": "…", + "projectId": "…", + "storageBucket": "…", + "messagingSenderId": "…", + "appId": "…", + "databaseURL": "…" + }, + "firebaseNotificationsDatabaseURL": "https://-notifications.firebaseio.com", + "cloudFunction": { + "validateClient": "https://…run.app", + "sdkProxy": "https://…/v2/core/a", + "setEncryptedData": "https://…", + "getNotificationsForDocuments": "https://…", + "getPlanDetails": "https://…", + "getAllowedDocuments": "https://…", + "sa": "https://…" + // Module-gated keys added only when provisioned, e.g.: + // "chatgptCompletion", "getIceServers", "whisperTranscription" + } + // "dataRegions": ["europe-west1"] // only if tfvars set it — mirror verbatim +} +``` + +### Semantics + +| Field | Behavior | +| --- | --- | +| `strict: true` | Unspecified endpoints resolve to `velt://self-hosted-disabled/` (zero egress). Missing **core** endpoints are reported once via `console.error`. | +| `strict: false` / omitted | Unspecified endpoints fall back to the SDK build's SaaS defaults, which is **not** acceptable for full self-hosting. | +| `deploymentProfile` | Must be the resolved `enabledModules` array. Endpoints for modules absent from this list are inert even if a URL were injected. | +| `cloudFunction.*` | Absolute base URLs (Cloud Run). No `*.cloudfunctions.net` assumption. | +| `cloudFunction.setEncryptedData` | Single URL collapses the SaaS multi-region split into one endpoint on self-host. | +| `firebaseConfig` | Merged over SDK env defaults (injected keys win). Use the Terraform-emitted web app config, **not** the console-patched copy whose `authDomain` was rewritten for the console host. | + +### App wiring + +```ts +config: { + proxyDomain: 'https://static.example.com', // origin only + version: '', // path /lib/sdk@/velt.js + selfHosted: { /* object above */ } +} +``` + +CDN serving requirements (CORS, MIME, path shape): see the "Wire your app" step in [Get Started on GCP](/self-hosting/full/gcp/overview). + +--- + +## Console runtime config + +Self-host console builds load **same-origin** `/velt-console-config.json` before bootstrap. Terraform emits this file; the install guide deploys it next to the static bundle. + +Minimum validity (fetch must be 200 + JSON with non-empty): + +- `firebaseConfig.apiKey` +- `firebaseConfig.projectId` +- `firebaseConfig.appId` +- `firebaseConfig.databaseURL` + +Also carries endpoint overrides (`sendLoginLink`, `aiChat`, and others), `enabledModules`, and self-host flags. Fail-soft: if the fetch fails or the payload is invalid, the app boots on build defaults and logs one error. It never falls open to Velt SaaS URLs (self-host builds ship with those stripped). + +`configSchemaVersion` inside the bundle's `console-version.json` must match the backend-emitted schema (pinned in the umbrella manifest as `console.configSchemaVersion`). + +--- + +## State file contract + +`velt-selfhost-state.json` (working directory or config repo) is the resume and handoff record for Install/Upgrade. + +```jsonc +{ + "guideVersion": 1, + "release": { /* full verified manifest */ }, + "inputs": { + "projectId": "", + "region": "us-central1", + "profile": "core", + "optInModules": [], + "ownerEmail": "", + "consoleBase": "", + "cdnBase": "" + }, + "phases": { + "0-preflight": "pending|done", + "1-project-bootstrap": "pending|in-progress|done", + "2-backend": "…", + "3-workspace": "…", + "4-console": "…", + "5-sdk-cdn": "…", + "6-acceptance": "…" + }, + "artifacts": { + "moduleDir": "", + "imageRef": "", // customer-registry ref, pinned BY DIGEST + "serviceUrls": {}, + "bootstrapResultPath": "", // SECRET — path only + "selfHostedConfigPath": "", + "consoleUrl": "", + "sdkCdnPath": "" + }, + "humanSteps": { + "billing": "pending|done", // link a billing account on the GCP project + "oauthClient": "pending|done", + "infosecScan": "pending|done|waived", + "dnsRecord": "pending|done|n/a", + "signInTest": "pending|done" + }, + "log": [{ "ts": "…", "phase": "…", "note": "…" }] +} +``` + +Store secret **values** in your secret manager; record only paths and references here. On upgrade, replace `release` only after the upgrade completes so an interrupted run still knows the installed version. + +--- + +## Backend module archive contents + +Pulled from `backend.moduleRef`, verified with `backend.moduleSha256`: + +| Path | Purpose | +| --- | --- | +| `terraform/` | Blueprint (no local state / emitted artifacts) | +| `functions/lib/` | Compiled deployment-profiles CLI for Terraform `external` data. Node ≥ 20, no `npm install` on the customer side | + +Install/Upgrade guides are **not** packed into the archive. Use the copies on this docs site. + +--- + +## Acceptance criteria (definition of done) + +A deployment is complete only when all hold: + +1. Console sign-in as a seeded admin lands on the dashboard with the workspace + keys. +2. App loads `velt.js` + chunks from **your** CDN (nothing from `cdn.velt.dev`); `window.Velt.version` matches the pin. +3. Creating a comment in the app persists and appears in the console data browser. +4. Network audit on app + console sessions: **no** requests to `velt.dev` or other Velt-owned hosts. + +--- + +## Related documents + +| Document | Role | +| --- | --- | +| [Overview](/self-hosting/full/overview) | Concepts and architecture | +| [Get Started](/self-hosting/full/gcp/overview) | The short path for a first deployment | +| [Install](/self-hosting/full/gcp/install) | Greenfield procedure | +| [Upgrade](/self-hosting/full/gcp/upgrade) | Delta upgrade procedure | diff --git a/self-hosting/full/gcp/upgrade.mdx b/self-hosting/full/gcp/upgrade.mdx new file mode 100644 index 000000000..53801b948 --- /dev/null +++ b/self-hosting/full/gcp/upgrade.mdx @@ -0,0 +1,294 @@ +--- +title: "Upgrade Guide" +sidebarTitle: "Upgrade" +description: "Agent-executable procedure for moving an existing Velt full self-hosting deployment to a newer umbrella release. Computes the component delta and runs only the tracks that changed." +--- + + + This guide is written to be executed by an AI coding agent with shell access. Hand it the page and let it work. A human can also follow it manually, since every step is a plain command. Setting up a deployment for the first time? Use the [Install guide](/self-hosting/full/gcp/install) instead. + + +**Doc set.** Human orientation and concepts: [Overview](/self-hosting/full/overview). Field-level contracts (manifest, profiles, configs, trust): [Reference](/self-hosting/full/gcp/reference). Greenfield install: [Install guide](/self-hosting/full/gcp/install). + +**Audience:** an AI coding agent with shell access, working for a Velt customer that already has a running self-hosted deployment (installed via the [Install guide](/self-hosting/full/gcp/install)). + +**Outcome:** the deployment moved to a newer self-host release (backend image + Terraform module, console bundle, and/or SDK files) with signature verification, scan sign-off, and rollback anchors at every step. Components upgrade independently: a release that only changes the backend touches nothing else. + +**Prerequisite:** the install's `velt-selfhost-state.json` (and the tfvars + Terraform state from Phase 2). If the state file is lost, reconstruct it first: the installed release version is in the deployed manifest recorded at install time, `terraform output` reproduces the service URLs, and the pinned SDK version is readable from the app's `lib/sdk@/` path. Do not guess, because an upgrade computed against the wrong installed version applies the wrong delta. + +--- + +## To the AI agent, read this first + +- **The upgrade is a delta, not a reinstall.** Phase U0 computes which components changed; you run only the tracks whose pins changed. Most releases are backend-only, and that is the expected common case, not a shortcut. +- **Backend image and Terraform module upgrade together, always.** Every Cloud Run service runs the same image with `FUNCTION_TARGET` selecting the export; the module at a given version knows which function set to deploy. Splitting them can point a service at an export that no longer exists. The signed manifest binds `backend.digest` and `backend.moduleRef`, so take both or neither. +- **Roll forward is the primary recovery strategy.** Rollback anchors exist at every step (old image digest stays in the customer registry, old module dir plus a pre-upgrade Terraform state backup are kept, old SDK folder is never deleted), but a backend rollback re-applies an older module against a state the newer module already mutated. Treat it as an emergency move, not a routine one. +- **Maintain the state file** after every step, same contract as the install guide. Log each upgrade in `log` and replace `release` only at the very end (U4), so an interrupted upgrade resumes knowing the old release is still the one of record. +- **Never run commands against any project other than `$PROJECT_ID`.** +- The install guide's retryable-error list, shell quirks (zsh `${VAR}:` bracing), and Troubleshooting appendix all apply here unchanged. + +--- + +## Phase U0: Resolve the target release and compute the delta + +### U0.1 Fetch and verify the target manifest + +```bash +RELEASE_REGISTRY="us-docker.pkg.dev/velt-sdk/velt-releases" + +# Enumerate releases (newest first) and pick the target with the operator — +# default to the newest, but the operator may pin an intermediate version. +oras pull -o . "$RELEASE_REGISTRY/velt-selfhost-index:latest" # writes index.json +python3 -c "import json;[print(r['selfHostVersion'], r['releasedAt']) for r in json.load(open('index.json'))['releases']]" + +TARGET_VERSION="" +oras pull -o . "$RELEASE_REGISTRY/velt-selfhost-manifest:$TARGET_VERSION" # writes manifest.json + +# MANDATORY: verify the manifest signature (same rule as install Phase 0.1). +# Only deploy releases that verify successfully. +MANIFEST_DIGEST=$(oras manifest fetch --descriptor "$RELEASE_REGISTRY/velt-selfhost-manifest:$TARGET_VERSION" | python3 -c "import json,sys;print(json.load(sys.stdin)['digest'])") +cosign verify \ + --certificate-identity-regexp 'https://github\.com/[^/]+/shared-firebase-function/.*' \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com \ + "$RELEASE_REGISTRY/velt-selfhost-manifest@$MANIFEST_DIGEST" +``` + +**Read `releaseNotes` before anything else.** The current release contract is **migration-free**: a release never requires a customer-side data migration, because new backend code reads existing data shapes. If a future release ever breaks this rule, its release notes and manifest will say so explicitly and ship a dedicated migration runbook. In that case STOP here and follow that runbook's ordering instead of this guide's. + +### U0.2 Compute the component delta + +Compare the target manifest against the installed one (`release` in the state file): + +```bash +python3 - <<'EOF' +import json +new = json.load(open('manifest.json')) +old = json.load(open('velt-selfhost-state.json'))['release'] +def row(name, o, n): print(f"{name:10} {'CHANGED' if o != n else 'same':8} {o} -> {n}" if o != n else f"{name:10} same {o}") +row('backend', old['backend']['digest'], new['backend']['digest']) +row('module', old['backend']['moduleSha256'], new['backend']['moduleSha256']) +row('console', old['console']['version'], new['console']['version']) +row('sdk', old['sdk']['testedVersion'], new['sdk']['testedVersion']) +EOF +``` + +| Delta | Upgrade track to run | +|---|---| +| `backend` or `module` changed | **U1** (they always change together; a mismatch where only one changed is a malformed release, so stop and contact Velt) | +| `console` changed | **U2** | +| `sdk` changed | **U3** (operator may defer, see the fence below) | +| Nothing changed | You're done, the target is already installed | + +Always finish with **U4** (re-acceptance and state file update) regardless of which tracks ran. + +### U0.3 Compatibility fences (hard gates, check BEFORE touching anything) + +**Fence 1: deployed SDK vs the target's `sdk.minVersion`.** A backend-only upgrade is safe only while the SDK the customer's app currently pins is inside the new release's supported window: + +```bash +DEPLOYED_SDK="" +MIN_SDK=$(python3 -c "import json;print(json.load(open('manifest.json'))['sdk']['minVersion'])") +# sort -V: the min version must sort first (or equal) for the fence to pass +[ "$(printf '%s\n%s\n' "$MIN_SDK" "$DEPLOYED_SDK" | sort -V | head -1)" = "$MIN_SDK" ] \ + && echo "SDK FENCE: PASS" || echo "SDK FENCE: FAIL — U3 is MANDATORY in this upgrade" +``` + +If the fence fails, the SDK track (U3) is no longer optional: plan the backend apply and the app's SDK repoint as one coordinated window with the operator. Note `sdk.testedVersion` too, because a deployed SDK above `minVersion` but below `testedVersion` is *supported but untested*. Recommend U3 in that case, don't force it. + +**Fence 2: console config schema.** The new Terraform module re-emits `velt-console-config.json` on apply. The deployed console bundle must support the schema the new module emits: + +```bash +python3 -c "import json;print('console configSchemaVersion:', json.load(open('manifest.json'))['console']['configSchemaVersion'])" +# Compare against the installed console's configSchemaVersion (state file release.console). +# Different → U2 is MANDATORY and must complete before or immediately after U1's apply. +``` + +**Fence 3: manifest schema.** `schemaVersion` must be one this guide understands (currently `2`). A higher number means the docs have moved on since your copy of this guide was saved, so re-fetch the current guide from the Velt docs site before proceeding. + +### U0.4 Preflight + +- Tooling: same table as install Phase 0.4 (`cosign`, `oras`, `crane`/docker, `terraform`, `gcloud`, `node`, `python3`) and the same three auth probes. +- **Back up before any mutation:** copy `velt-selfhost-state.json`, the tfvars file, and the Terraform state (`terraform state pull > tfstate-pre-${TARGET_VERSION}.json` from the *old* module dir) somewhere outside the working tree. +- Show the operator the human-moment table for this upgrade: infosec scan sign-off (if U1 runs and org policy requires it, roughly 5 to 15 minutes), and the app team's SDK repoint (only if U3 runs). There is no billing, OAuth, or DNS moment in an upgrade, because all of that infrastructure survives untouched. + +**Verify (phase gate):** target manifest verified and stored; delta table computed and agreed with the operator; all three fences pass (or their mandatory tracks are scheduled); backups taken. Record `upgrade: { from, to, tracks }` in the state file log. + +--- + +## Phase U1: Backend upgrade (image + module, one track) + +### U1.1 Copy and verify the new image + +Same mechanics as install Phase 1.4: digest-preserving copy into the customer's own registry, then cosign-verify the copy. + +```bash +NEW_DIGEST=$(python3 -c "import json;print(json.load(open('manifest.json'))['backend']['digest'])") +SRC=$(python3 -c "import json;print(json.load(open('manifest.json'))['backend']['imageByDigest'])") +DEST="us-docker.pkg.dev/$PROJECT_ID/velt/velt-functions" # the SAME dest repo the install used + +crane copy "$SRC" "$DEST:${TARGET_VERSION}" # (docker pull/tag/push fallback: install 1.4 — remember --platform linux/amd64 on arm hosts) +crane digest "$DEST:${TARGET_VERSION}" # MUST equal $NEW_DIGEST +cosign verify \ + --certificate-identity-regexp 'https://github\.com/[^/]+/shared-firebase-function/.*' \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com \ + "$DEST@$NEW_DIGEST" # MUST print Verified OK — STOP otherwise +NEW_IMAGE_REF="$DEST@$NEW_DIGEST" # pin BY DIGEST, as always +``` + + + Do **not** delete or retag the currently-deployed image. It is the backend rollback anchor. Both digests coexist in the customer registry. + + +### U1.2 Scan the new image (operator's infosec policy, human moment) + +Identical to install Phase 1.5: scan the copy, compare CRITICAL/HIGH findings against the **new** manifest's `backend.knownFindings`. Findings **in** the list are Velt-accepted (unfixable base-OS CVEs); findings **not** in the list are a stop-and-contact-Velt condition. The operator signs off (or explicitly waives scanning) before the apply. + +### U1.3 Extract the new module next to the old one, migrate tfvars and state + +```bash +MODULE_REF=$(python3 -c "import json;print(json.load(open('manifest.json'))['backend']['moduleRef'])") +MODULE_SHA=$(python3 -c "import json;print(json.load(open('manifest.json'))['backend']['moduleSha256'])") +oras pull -o . "$MODULE_REF" +MODULE_TAR=$(ls velt-backend-module-${TARGET_VERSION}.tar.gz) +echo "$MODULE_SHA $MODULE_TAR" | shasum -a 256 -c - # MUST print OK + +OLD_MODULE_DIR="" +export MODULE_DIR="$(pwd)/velt-backend-module-${TARGET_VERSION}" +mkdir -p "$MODULE_DIR" && tar -xzf "$MODULE_TAR" -C "$MODULE_DIR" + +# The Terraform STATE and tfvars are the deployment's memory — carry them over. +# (If the operator configured a remote state backend at install time, skip the +# state copy; init below will find it. The tfvars copy is always needed.) +cp "$OLD_MODULE_DIR/terraform/velt.auto.tfvars" "$MODULE_DIR/terraform/" +cp "$OLD_MODULE_DIR/terraform/terraform.tfstate"* "$MODULE_DIR/terraform/" 2>/dev/null || true + +cd "$MODULE_DIR/terraform" +# Point the image pin at the new digest: +# velt_image = "" +# Leave every other tfvars value alone unless the new module README says otherwise. +terraform init && terraform validate +``` + + + Keep `$OLD_MODULE_DIR` intact until U4 passes. It is the module-side rollback anchor. No Identity Platform re-import is needed, because the resource is already in the carried-over state. + + +### U1.4 Plan review, the upgrade's delta detector + +```bash +terraform plan -out=upgrade.plan +``` + +Read the plan, don't skim it. Expected shape: in-place updates and replacements of Cloud Run services (new image), possibly new resources the new module version adds (new secrets, new services, new indexes). Red flags, STOP and investigate before applying: + +- **Destroy of a data-bearing resource** (Firestore database, RTDB instance, storage bucket). No upgrade should ever destroy one. +- Resources outside `$PROJECT_ID`. +- A destroy/create of *every* service, which suggests the state didn't carry over. You are about to duplicate the stack, not upgrade it. + +### U1.5 Secrets delta (the seed-before-apply rule from install 2.4 still binds) + +```bash +# New module versions may ADD entries to secrets_to_seed. Cloud Run refuses a +# revision mounting a secret with no enabled version — seed NEW ids before the apply. +terraform apply -target=google_secret_manager_secret.required \ + -target=google_secret_manager_secret_iam_member.runtime_accessor \ + -target=google_secret_manager_secret_version.optional_placeholder +terraform output -json secrets_to_seed +# Diff against what exists: any id in the output with no enabled version needs seeding. +for id in $(terraform output -json secrets_to_seed | python3 -c "import json,sys;print(' '.join(json.load(sys.stdin)))"); do + gcloud secrets versions list "$id" --project="$PROJECT_ID" --filter="state=enabled" --limit=1 --format="value(name)" | grep -q . \ + || echo "NEEDS SEEDING: $id" +done +# Seed per the install guide 2.4 value rules (real keys for enabled AI modules, +# "placeholder-not-configured" for listed-but-unused, fresh random for crypto keys — +# NEVER regenerate an EXISTING crypto secret: JWT_SECRET_KEY / PLUGIN_CRYPTO_* rotate +# only via a deliberate, separately-planned rotation, not as an upgrade side effect). +``` + +### U1.6 Apply, converge, and smoke + +```bash +terraform apply upgrade.plan # rolling Cloud Run revisions — near-zero downtime +terraform plan # MUST end "No changes." (retryables: install guide list) +terraform output -json service_urls > service-urls.json +``` + +Re-run the install guide's Phase 2.6 smoke tests (callable envelope on `validateclient`, HTTP 400 on empty `cacheddata` POST). Then watch the logs for roughly 10 minutes of normal traffic for `FAILED_PRECONDITION` composite-index errors, because a new release may query patterns the deployed indexes don't cover yet. If seen: add the index to `default_db_composite_indexes` in tfvars and re-apply (default DB), and report it to Velt (Velt uses the report to keep indexes complete in future releases; workspace-DB gaps need a Velt-shipped fix). + +**Verify (track gate):** plan converged; all services Ready=True; smokes pass; the serving revisions reference `$NEW_DIGEST` (`gcloud run services describe --format="value(spec.template.spec.containers[0].image)"`, spot-check 2 or 3 services). Update `artifacts.imageRef` and `artifacts.moduleDir` in the state file. + +**Rollback (emergency only):** repoint `velt_image` to the old digest and run the apply from `$OLD_MODULE_DIR/terraform` with the pre-upgrade state backup restored (`terraform state push tfstate-pre-${TARGET_VERSION}.json`, and coordinate with Velt if the new module already created resources the old one doesn't know). Prefer rolling forward to a fixed release. + +--- + +## Phase U2: Console upgrade (only if the console pin changed) + +```bash +BUNDLE_REF=$(python3 -c "import json;print(json.load(open('manifest.json'))['console']['bundleRef'])") +CONSOLE_SHA=$(python3 -c "import json;print(json.load(open('manifest.json'))['console']['sha256'])") +oras pull -o . "$BUNDLE_REF" +echo "$CONSOLE_SHA $(ls console-dist-*.tar.gz)" | shasum -a 256 -c - # MUST print OK +``` + +Stage and deploy exactly as install Phase 4.2, to the **same host** the install chose, with two upgrade-specific rules: + +1. **Preserve the runtime config.** `velt-console-config.json` at the web root is deployment-specific (it was emitted by Terraform, with `authDomain` patched to the console's own domain). Deploy the new bundle files, then re-place the existing config. If U1 ran and re-emitted a fresh config, re-apply the `authDomain` check from install 4.2 before uploading it. +2. **Bust the SPA cache.** After deploy, a hard refresh must show the new build. Verify `curl $CONSOLE_BASE/velt-console-config.json` still returns the valid config, then sign in and load any data view. + +**Verify (track gate):** console loads, sign-in works, a data view renders. Update the `artifacts.consoleUrl` bundle version note in the state file. Rollback: redeploy the previous bundle (keep the old `console-dist-*.tar.gz` until U4 passes). + +--- + +## Phase U3: SDK upgrade (only if the SDK pin changed, or Fence 1 failed) + +The SDK ships as static files under a **versioned folder**, so upgrades are additive by design: + +```bash +SDK_REF=$(python3 -c "import json;print(json.load(open('manifest.json'))['sdk']['bundleRef'])") +SDK_SHA=$(python3 -c "import json;print(json.load(open('manifest.json'))['sdk']['sha256'])") +SDK_VERSION=$(python3 -c "import json;print(json.load(open('manifest.json'))['sdk']['testedVersion'])") +oras pull -o . "$SDK_REF" +echo "$SDK_SHA $(ls veltdev-sdk-*.tgz)" | shasum -a 256 -c - # MUST print OK +``` + +1. Upload the package contents to a **NEW** `lib/sdk@${SDK_VERSION}/` folder on the same CDN or static host, with the same serving metadata as install Phase 5 (`Content-Type: text/javascript`, ACAO header, cache-control). **Never overwrite or delete the currently-pinned folder**, because it is the instant rollback. +2. Verify: `curl -sI "$CDN_BASE/lib/sdk@${SDK_VERSION}/velt.js"` returns 200 with the correct content-type and ACAO. +3. **Hand the repoint to the app team** (this is their change, not yours): bump the `@veltdev/*` wrapper packages to `${SDK_VERSION}` **and** the `version` pin in the app's `selfHosted` config in the same deploy. A wrapper/pin mismatch is the "works, then breaks after an npm upgrade" failure in the install guide's troubleshooting table. Rollback is repointing the pin to the old folder. + +**Verify (track gate):** new folder serves correctly; the app team has the repoint instructions (or has completed them, if this upgrade is a coordinated window from Fence 1). Update `artifacts.sdkCdnPath` when the repoint lands. + +--- + +## Phase U4: Re-acceptance and record the upgrade + +Run the subset of install Phase 6 matching the tracks that ran: + +- **U1 ran:** demo-app comment loop against the upgraded backend (create a comment, it renders, API calls go to the customer's `*.run.app` services) plus no new errors in service logs. +- **U2 ran:** the comment or data from the loop above is visible in the console's data browser. +- **U3 ran (and repoint landed):** `window.Velt.version` equals the new pinned version. +- **Always:** zero-egress spot check, with no requests to any `velt.dev` host in the browser network log. + +Then close the books: + +```bash +# State file: replace `release` with the new manifest, keep the old one on disk +mv manifest.json "manifest-${TARGET_VERSION}.json" +# update velt-selfhost-state.json: release = new manifest, log += upgrade record +``` + +Report to the operator: versions moved (from and to, per component), scan sign-off outcome, anything deferred (for example an SDK repoint pending on the app team), and the rollback anchors now in place (old image digest, `$OLD_MODULE_DIR` plus state backup, old SDK folder). After a soak period the operator is comfortable with (suggest at least 1 week), the old module dir and pre-upgrade state backup can be archived and the old image untagged, never before. + +--- + +## Upgrade troubleshooting (deltas from the install guide's table) + +| Symptom | Cause | Fix | +|---|---|---| +| Plan wants to destroy/create every service | Terraform state didn't carry into the new module dir | Stop; copy `terraform.tfstate*` from the old module dir (U1.3), re-init | +| Apply fails mounting a secret | New `secrets_to_seed` id not seeded | U1.5 diff loop; seed, re-apply | +| Services Ready but errors on one endpoint post-upgrade | `FAILED_PRECONDITION` composite-index gap in the new release | U1.6: add to `default_db_composite_indexes` (default DB) / report to Velt (workspace DB) | +| Console blank or auth-broken after U2 | `velt-console-config.json` overwritten by the bundle deploy, or `authDomain` regressed to `.firebaseapp.com` | Re-place the preserved config; re-run the install 4.2 authDomain check | +| App breaks right after U3 repoint | Wrapper version and CDN pin moved separately | Both move in the same app deploy; rollback = repoint the pin | +| Old SDK erroring against new backend | Deployed SDK below the new `sdk.minVersion` (Fence 1 skipped) | Complete U3 and repoint now; the fence exists to catch this before the apply | + +*This guide is evergreen: every version-specific value comes from the signed release manifest the operator selects in Phase U0, so the guide text itself does not change per release. Pairs with the [Install guide](/self-hosting/full/gcp/install).* diff --git a/self-hosting/full/overview.mdx b/self-hosting/full/overview.mdx new file mode 100644 index 000000000..4f2ab7e46 --- /dev/null +++ b/self-hosting/full/overview.mdx @@ -0,0 +1,128 @@ +--- +title: "Overview" +description: "Run the complete Velt platform on infrastructure you own: backend, admin console, and client SDK, with no runtime requests to Velt-owned hosts." +--- + +Velt's SaaS product runs on Velt-operated GCP: Cloud Functions, Firestore/RTDB, the hosted console at `console.velt.dev`, and the SDK from `cdn.velt.dev`. **Full self-hosting** moves that entire stack into a cloud project you own. + +| Piece | What you run | What you stop calling | +| --- | --- | --- | +| **Backend** | Signed container on Cloud Run (Terraform), your Firebase project, your secrets | `api.velt.dev`, `*.cloudfunctions.net` on Velt projects | +| **Console** | Static admin UI on your host, config pointing at your backend | `console.velt.dev` | +| **SDK** | `@veltdev/sdk` files on your CDN + `config.selfHosted` in the app | `cdn.velt.dev` for code; Velt backends for data | + +When finished, your app and admin surface make **no** requests to `velt.dev` or Velt-owned hosts. Data, auth, and admin stay inside your tenancy. + +You consume releases through a single signed **umbrella manifest** (`velt-selfhost-manifest`) that pins backend image + Terraform module, console bundle, and the SDK version this release was tested with. You never hand-pick mismatched component versions. + +## Full vs partial self-hosting + +Velt offers two different things under the word "self-hosting", and they solve different problems. + +**Partial self-hosting** keeps Velt's managed backend and moves only your user-generated content and PII into your own storage. You register data providers, Velt stores the structural non-PII data (IDs, document and organization references, locations, statuses, timestamps), and your backend stores the content. It is an application-level integration with no infrastructure to run. + +**Full self-hosting** moves the entire stack. There is no Velt-operated component left in the runtime path. + +| | Partial self-hosting | Full self-hosting | +| --- | --- | --- | +| **What moves to you** | User content and PII, through data providers | Backend, console, SDK hosting, and all data | +| **Who runs the backend** | Velt | You, on your own GCP project | +| **What you set up** | Data provider callbacks or endpoints in your app | GCP project, Terraform, Firebase, console host, CDN | +| **Requests to Velt hosts** | Yes, for the collaboration backend | None | +| **Where to start** | [Partial self-hosting](/self-hosting/partial/overview) | [Get Started on GCP](/self-hosting/full/gcp/overview) | + +## Core concepts + +| Concept | What it is | +| --- | --- | +| **Umbrella release** | A versioned binding of backend + console + SDK, published as `velt-selfhost-manifest:` (and `:latest`). One number (`selfHostVersion`) is what you install and upgrade. | +| **Signed manifest** | Cosign-signed JSON. Trust root for the release: it pins image digest, module archive sha256, console sha256, SDK sha256. Always verify by digest, never trust a mutable tag alone. | +| **Backend image** | One container (`velt-functions`) serving many Cloud Run services; `FUNCTION_TARGET` selects the export. Image and Terraform **module always upgrade together**. | +| **Deployment profile** | Curated feature set (`core`, `core+recording`, `core+ai+agents`, `full`) plus optional opt-in modules. Terraform provisions only that closure. | +| **`enabledModules`** | Resolved module list written into `velt-deployment-profile.json`. Console and SDK both honor it, hiding or inert-sentinelling features that aren't deployed. | +| **`config.selfHosted`** | App-side object that repoints every SDK endpoint at your deployment. Generated during install, so don't hand-type URLs. | +| **Strict mode** | `selfHosted.strict: true`. Any endpoint you didn't inject resolves to an inert `velt://self-hosted-disabled/…` sentinel instead of falling back to Velt SaaS. Required for zero-egress. | +| **State file** | `velt-selfhost-state.json`, the resume contract for install/upgrade agents and humans. Records inputs, phase status, and artifact paths (not secret values). | + +## Architecture + +```mermaid +flowchart TB + subgraph Customer["Your GCP project"] + App["Your app
@veltdev/react + selfHosted"] + CDN["Your CDN
/lib/sdk@version/"] + Console["Your console host
velt-console-config.json"] + CR["Cloud Run services
velt-functions image"] + FB["Firebase
Auth · Firestore · RTDB · Storage"] + end + + Registry["Velt release registry
velt-selfhost-manifest
image · module · console · sdk"] + + App -->|loads JS| CDN + App -->|callables / data| CR + App -->|Auth + listeners| FB + Console -->|callables| CR + Console -->|Auth + admin data| FB + CR --> FB + Registry -.->|install / upgrade pulls| Customer +``` + +**Invariant:** after acceptance, DevTools Network on the app and the console show **no** requests to `cdn.velt.dev`, `console.velt.dev`, `api.velt.dev`, or other Velt-owned hosts. + +## What's included + +### Always on (`core`) + +Auth/identity, data plane, workspace control plane, notifications pipeline, agents runtime, analytics/debugger backends, console **backend** handlers, and index converge tooling. This is everything needed for the collaboration product plus the admin API on a self-hosted stack. + +### Curated add-ons (pick a profile or opt in) + +| Module | Typical need | +| --- | --- | +| `rest-api` | Customer-facing `*apibe` REST surface, GDPR workers, bulk folder moves | +| `recorder-media` | Recording / Whisper / screenshots | +| `ai` | AI chat / completion Cloud Functions (BYO LLM keys) | +| `huddle-webrtc` | Huddle / TURN (`getIceServers`) | +| `integrations-workflow` | Workflow engine + third-party connectors (opt-in) | +| `migrations` | Operational migration tooling (off by default; recommended for full console ops) | + +Exact profile compositions: [Reference, Deployment profiles](/self-hosting/full/gcp/reference#deployment-profiles). + +Features whose modules aren't in `enabledModules` are gated in the console UI and inert-sentinelled in the SDK under strict mode. Self-host console builds also strip SaaS-only telemetry. + +## Security and trust + +1. **Verify the manifest** with `cosign` against the GitHub Actions OIDC identity before any pull of image, module, console, or SDK pins. +2. **Pin the image by digest** in your registry and in Terraform. Tags move; digests don't. +3. **Re-scan the image** in your tenancy (Phase 1.5) and diff against the manifest's signed `knownFindings`. Fixable CRITICAL/HIGH findings are never "accepted". +4. **Secrets:** crypto keys (`PLUGIN_CRYPTO_*`, `JWT_SECRET_KEY`) plus your Gemini and Anthropic API keys must be real values you control (required even on `core`). Optional Twilio and other BYO keys apply when those modules are enabled. Velt SaaS analytics/OAuth placeholders are auto-seeded so Cloud Run can start; replace OAuth placeholders with *your* app credentials before enabling an integration. +5. **Zero-egress proof** is part of acceptance. Treat residual `velt.dev` calls as a failed install. + +Details: [Reference, Trust model](/self-hosting/full/gcp/reference#trust-model). + +## Limitations + +- **GCP + Firebase only.** Portable and non-Firebase backends are out of scope today. +- **Human steps remain:** link a billing account on the GCP project, create the OAuth client, sign off the infosec scan, set up DNS (if custom domains), and complete the first console sign-in. +- **Install and upgrade guides are evergreen.** Version-specific pins always come from the signed manifest at run time, so don't bake component versions into runbooks. + +## Supported clouds + +| Cloud | Status | Guide | +| --- | --- | --- | +| GCP + Firebase | Available | [Get Started on GCP](/self-hosting/full/gcp/overview) | +| AWS | Closed beta | [AWS](/self-hosting/full/aws) | +| Azure | Closed beta | [Azure](/self-hosting/full/azure) | + +The concepts on this page apply to every cloud. The umbrella release, signed manifest, deployment profiles, and strict mode do not change per platform. + +## Next steps + + + + Prerequisites, the inputs you decide once, and how to hand the install guide to your coding agent. + + + Manifest schema, profiles, config shapes, and trust verification. + + diff --git a/self-host-data/activity.mdx b/self-hosting/partial/activity.mdx similarity index 99% rename from self-host-data/activity.mdx rename to self-hosting/partial/activity.mdx index 6c5e3dee2..47c24720f 100644 --- a/self-host-data/activity.mdx +++ b/self-hosting/partial/activity.mdx @@ -96,7 +96,7 @@ Both approaches are fully backward compatible and can be used together. Instead of implementing custom methods, you can configure endpoints directly and let the SDK handle HTTP requests. - `headers` may also be an async function resolved per request, and `credentials` (`'include' | 'same-origin' | 'omit'`) enables cookie/session auth — see [Async headers and credentials](/self-host-data/overview#async-headers-and-credentials) for details. + `headers` may also be an async function resolved per request, and `credentials` (`'include' | 'same-origin' | 'omit'`) enables cookie/session auth — see [Async headers and credentials](/self-hosting/partial/overview#async-headers-and-credentials) for details. diff --git a/self-host-data/attachments.mdx b/self-hosting/partial/attachments.mdx similarity index 99% rename from self-host-data/attachments.mdx rename to self-hosting/partial/attachments.mdx index 9ddbda2bb..d2ed62f1b 100644 --- a/self-host-data/attachments.mdx +++ b/self-hosting/partial/attachments.mdx @@ -57,7 +57,7 @@ Both approaches are fully backward compatible and can be used together. Instead of implementing custom methods, you can configure endpoints directly and let the SDK handle HTTP requests. - `headers` may also be an async function resolved per request, and `credentials` (`'include' | 'same-origin' | 'omit'`) enables cookie/session auth — see [Async headers and credentials](/self-host-data/overview#async-headers-and-credentials) for details. + `headers` may also be an async function resolved per request, and `credentials` (`'include' | 'same-origin' | 'omit'`) enables cookie/session auth — see [Async headers and credentials](/self-hosting/partial/overview#async-headers-and-credentials) for details. diff --git a/self-host-data/comments.mdx b/self-hosting/partial/comments.mdx similarity index 99% rename from self-host-data/comments.mdx rename to self-hosting/partial/comments.mdx index bebe0f1cd..9b266c4fb 100644 --- a/self-host-data/comments.mdx +++ b/self-hosting/partial/comments.mdx @@ -256,7 +256,7 @@ Velt.setDataProviders({ comment: commentDataProvider }); ### Async headers and credentials -Endpoint configs (`getConfig`, `saveConfig`, `deleteConfig`) support an async `headers` function for short-lived per-request tokens, and a `credentials` field for cookie/session auth. This applies to all endpoint-based providers — see [Async headers and credentials](/self-host-data/overview#async-headers-and-credentials) for details and examples. +Endpoint configs (`getConfig`, `saveConfig`, `deleteConfig`) support an async `headers` function for short-lived per-request tokens, and a `credentials` field for cookie/session auth. This applies to all endpoint-based providers — see [Async headers and credentials](/self-hosting/partial/overview#async-headers-and-credentials) for details and examples. ### additionalSaveEvents @@ -628,11 +628,11 @@ Configuration for the comment data provider. - `saveRetryConfig`: [`RetryConfig`](/api-reference/sdk/models/data-models#retryconfig). Configure retry behavior for save operations. - `deleteRetryConfig`: [`RetryConfig`](/api-reference/sdk/models/data-models#retryconfig). Configure retry behavior for delete operations. - `additionalFields`: `string[]`. Custom fields that are **copied** into the resolver request payloads sent to your backend while still being **kept** in Velt's database. Use this when you want a copy of a field in your own backend (e.g., for analytics or search) without removing it from Velt — for example `status`, `assignedTo`, `priority`. - - `fieldsToRemove`: `string[]`. Custom fields that are **moved** out of Velt's database into yours: they are sent to your backend, deleted from Velt's storage, and merged back on read. Use this for custom fields that must not be stored by Velt at all (e.g., `internalTicketId`). See [Excluding & extending fields](/self-host-data/overview#excluding--extending-fields) for the full behavior comparison. + - `fieldsToRemove`: `string[]`. Custom fields that are **moved** out of Velt's database into yours: they are sent to your backend, deleted from Velt's storage, and merged back on read. Use this for custom fields that must not be stored by Velt at all (e.g., `internalTicketId`). See [Excluding & extending fields](/self-hosting/partial/overview#excluding--extending-fields) for the full behavior comparison. - `additionalSaveEvents`: [`CommentResolverSaveEvent[]`](/api-reference/sdk/models/data-models#commentresolversaveevent). Additional non-core comment events to send to your save resolver, such as status changes, priority changes, assignment changes, approvals, comment-level reactions, and subscription changes. -`fieldsToRemove` is only for your own custom fields. Never list a field Velt uses to query, scope, position, sync, or render an annotation (e.g., `metadata`, `annotationId`, `status`, `location`, `from`, `isCommentResolverUsed`) or comments will fail to load, appear in the wrong place, or break filtering. If in doubt, use `additionalFields` instead. See the [full list of fields to avoid](/self-host-data/overview#excluding--extending-fields). +`fieldsToRemove` is only for your own custom fields. Never list a field Velt uses to query, scope, position, sync, or render an annotation (e.g., `metadata`, `annotationId`, `status`, `location`, `from`, `isCommentResolverUsed`) or comments will fail to load, appear in the wrong place, or break filtering. If in doubt, use `additionalFields` instead. See the [full list of fields to avoid](/self-hosting/partial/overview#excluding--extending-fields). ```jsx diff --git a/self-host-data/field-inventory.mdx b/self-hosting/partial/field-inventory.mdx similarity index 99% rename from self-host-data/field-inventory.mdx rename to self-hosting/partial/field-inventory.mdx index f960b7d5a..17a7675e1 100644 --- a/self-host-data/field-inventory.mdx +++ b/self-hosting/partial/field-inventory.mdx @@ -3,7 +3,7 @@ title: "Complete Field Inventory" description: "An exhaustive, ground-truthed inventory of every persisted self-hosting field — Velt's DB vs. your DB — with type, example value, description, and notes for comments, reactions, recordings, notifications, activity, and attachments." --- -> **Companion to the [Self-Hosting Overview](/self-host-data/overview).** That guide's field-inventory +> **Companion to the [Self-Hosting Overview](/self-hosting/partial/overview).** That guide's field-inventory > section lists the field *names* split between Velt's DB and your DB. This page adds **Type**, > **Example value**, **Description**, and **Notes** for every field, covers **both sides** of the > split, and **expands every nested structural object** into its own sub-table. Ground-truthed against diff --git a/self-host-data/notifications.mdx b/self-hosting/partial/notifications.mdx similarity index 99% rename from self-host-data/notifications.mdx rename to self-hosting/partial/notifications.mdx index 31a976039..61ec525eb 100644 --- a/self-host-data/notifications.mdx +++ b/self-hosting/partial/notifications.mdx @@ -59,7 +59,7 @@ Both approaches are fully backward compatible and can be used together. Instead of implementing custom methods, you can configure endpoints directly and let the SDK handle HTTP requests. - `headers` may also be an async function resolved per request, and `credentials` (`'include' | 'same-origin' | 'omit'`) enables cookie/session auth — see [Async headers and credentials](/self-host-data/overview#async-headers-and-credentials) for details. + `headers` may also be an async function resolved per request, and `credentials` (`'include' | 'same-origin' | 'omit'`) enables cookie/session auth — see [Async headers and credentials](/self-hosting/partial/overview#async-headers-and-credentials) for details. ### getConfig diff --git a/self-host-data/overview.mdx b/self-hosting/partial/overview.mdx similarity index 90% rename from self-host-data/overview.mdx rename to self-hosting/partial/overview.mdx index da84d81c7..10e5cb676 100644 --- a/self-host-data/overview.mdx +++ b/self-hosting/partial/overview.mdx @@ -25,13 +25,13 @@ For write requests (`save`, `delete`), the operation is performed on your databa | Feature | Provider key | `get` | `save` | `delete` | Model | |---|---|:---:|:---:|:---:|---| -| [Users](/self-host-data/users) | `user` | ✅ | — | — | Maps `userId → user object` | -| [Comments](/self-host-data/comments) | `comment` | ✅ | ✅ | ✅ | Strip on write, merge on read | -| [Reactions](/self-host-data/reactions) | `reaction` | ✅ | ✅ | ✅ | Strip on write, merge on read | -| [Attachments](/self-host-data/attachments) | `attachment` | — | ✅ | ✅ | File storage only | -| [Recordings](/self-host-data/recordings) | `recorder` | ✅ | ✅ | ✅ | Strip on write + file storage modes | -| [Activity](/self-host-data/activity) | `activity` | ✅ | ✅ | — | Append-only (no delete) | -| [Notifications](/self-host-data/notifications) | `notification` | ✅ | — | ✅ | Read-only enrichment (custom notifications only) | +| [Users](/self-hosting/partial/users) | `user` | ✅ | — | — | Maps `userId → user object` | +| [Comments](/self-hosting/partial/comments) | `comment` | ✅ | ✅ | ✅ | Strip on write, merge on read | +| [Reactions](/self-hosting/partial/reactions) | `reaction` | ✅ | ✅ | ✅ | Strip on write, merge on read | +| [Attachments](/self-hosting/partial/attachments) | `attachment` | — | ✅ | ✅ | File storage only | +| [Recordings](/self-hosting/partial/recordings) | `recorder` | ✅ | ✅ | ✅ | Strip on write + file storage modes | +| [Activity](/self-hosting/partial/activity) | `activity` | ✅ | ✅ | — | Append-only (no delete) | +| [Notifications](/self-hosting/partial/notifications) | `notification` | ✅ | — | ✅ | Read-only enrichment (custom notifications only) | | Anonymous users | `anonymousUser` | resolve-by-email | — | — | Maps `email → userId` | @@ -250,30 +250,30 @@ Every `get` / `save` / `delete` request includes a `metadata` object ([`BaseMeta ## Per-feature reference -Each provider has its own page with the full interface, request types, and both callback and endpoint examples. PII payload types are documented in [Data models](/api-reference/sdk/models/data-models); the field-by-field inventory of what gets stored where is in [Complete field inventory](/self-host-data/field-inventory). +Each provider has its own page with the full interface, request types, and both callback and endpoint examples. PII payload types are documented in [Data models](/api-reference/sdk/models/data-models); the field-by-field inventory of what gets stored where is in [Complete field inventory](/self-hosting/partial/field-inventory). | Provider | Methods | PII payload | Notes | |---|---|---|---| -| [`comment`](/self-host-data/comments) | `get` / `save` / `delete` | [`PartialCommentAnnotation`](/api-reference/sdk/models/data-models#partialcommentannotation) | Strip on write, merge on read. | -| [`reaction`](/self-host-data/reactions) | `get` / `save` / `delete` | [`PartialReactionAnnotation`](/api-reference/sdk/models/data-models#partialreactionannotation) | Strip on write, merge on read. | -| [`recorder`](/self-host-data/recordings) | `get` / `save` / `delete` (+ `storage`) | [`PartialRecorderAnnotation`](/api-reference/sdk/models/data-models#partialrecorderannotation) | Bring-your-own file storage via `storage`. | -| [`attachment`](/self-host-data/attachments) | `save` / `delete` | binary file → `{ url }` | File storage only; no `get`. | -| [`activity`](/self-host-data/activity) | `get` / `save` | [`PartialActivityRecord`](/api-reference/sdk/models/data-models#partialactivityrecord) | Append-only (no delete). May contain PII from other providers. | -| [`notification`](/self-host-data/notifications) | `get` / `delete` | [`PartialNotification`](/api-reference/sdk/models/data-models#partialnotification) | Read-only enrichment for `custom`-source notifications only. | -| [`anonymousUser`](/self-host-data/users#anonymous-user-resolution) | `resolveUserIdsByEmail` | `{ email: userId }` (transient) | Maps email → `userId` when @mentioning unknown users. | -| [`user`](/self-host-data/users) | `get` | `Record` | Resolves user objects for every other provider. | +| [`comment`](/self-hosting/partial/comments) | `get` / `save` / `delete` | [`PartialCommentAnnotation`](/api-reference/sdk/models/data-models#partialcommentannotation) | Strip on write, merge on read. | +| [`reaction`](/self-hosting/partial/reactions) | `get` / `save` / `delete` | [`PartialReactionAnnotation`](/api-reference/sdk/models/data-models#partialreactionannotation) | Strip on write, merge on read. | +| [`recorder`](/self-hosting/partial/recordings) | `get` / `save` / `delete` (+ `storage`) | [`PartialRecorderAnnotation`](/api-reference/sdk/models/data-models#partialrecorderannotation) | Bring-your-own file storage via `storage`. | +| [`attachment`](/self-hosting/partial/attachments) | `save` / `delete` | binary file → `{ url }` | File storage only; no `get`. | +| [`activity`](/self-hosting/partial/activity) | `get` / `save` | [`PartialActivityRecord`](/api-reference/sdk/models/data-models#partialactivityrecord) | Append-only (no delete). May contain PII from other providers. | +| [`notification`](/self-hosting/partial/notifications) | `get` / `delete` | [`PartialNotification`](/api-reference/sdk/models/data-models#partialnotification) | Read-only enrichment for `custom`-source notifications only. | +| [`anonymousUser`](/self-hosting/partial/users#anonymous-user-resolution) | `resolveUserIdsByEmail` | `{ email: userId }` (transient) | Maps email → `userId` when @mentioning unknown users. | +| [`user`](/self-hosting/partial/users) | `get` | `Record` | Resolves user objects for every other provider. | ## Complete field inventory For an exhaustive, ground-truthed breakdown of **every** persisted field — with **types**, **example values**, **descriptions**, and per-feature **strip rules** — covering both sides of the split (Velt's DB *and* your DB) and expanding every nested structural object into its own sub-table, see the dedicated reference: - + Every persisted field for comments, reactions, recordings, notifications, activity, and attachments — Velt's DB vs. your DB, with types, examples, and notes. A few rules apply across **all** features: -- **User objects are reduced to `{ userId }`** in Velt's DB for most features (comments, recordings, activity). The full user object (name, email, avatar, …) lives in your backend, resolved via your [`user`](/self-host-data/users) provider. It is the data source that all the other resolvers rely on when they strip user objects down to `{ userId }`. +- **User objects are reduced to `{ userId }`** in Velt's DB for most features (comments, recordings, activity). The full user object (name, email, avatar, …) lives in your backend, resolved via your [`user`](/self-hosting/partial/users) provider. It is the data source that all the other resolvers rely on when they strip user objects down to `{ userId }`. - **The client-facing `metadata`** (see above) accompanies every PII payload you store. - Velt sets an internal flag on the structural record (e.g., `isCommentResolverUsed`, `isReactionResolverUsed`, `isRecorderResolverUsed`, `isNotificationResolverUsed`, `isActivityResolverUsed`) so the UI knows to wait for resolver data. You don't need to store these. @@ -355,7 +355,7 @@ Your endpoint stores the file and responds `{ "data": { "url": "https://…" } } When a recording storage provider (`recorder.storage`) is set, Velt uploads the **entire recording** to your storage once — after the recording stops and the annotation is saved — and then patches the returned file URL onto the annotation. Velt also skips its own server-side encoding/transcription post-processing in this case; you own those files end to end. -See [Recordings](/self-host-data/recordings) and [Attachments](/self-host-data/attachments) for full examples. +See [Recordings](/self-hosting/partial/recordings) and [Attachments](/self-hosting/partial/attachments) for full examples. ## Excluding & extending fields @@ -447,7 +447,7 @@ After this, `priorityScore` is never written to Velt's database — it lives onl - **Retries.** `retryCount` / `retryDelay` let you retry transient failures. Combine with `revertOnFailure` to roll back Velt's optimistic local change if your save/delete ultimately fails. - **Degrade, don't drop.** If resolution fails, Velt keeps the structural record and renders it without the PII rather than dropping it. Your data is never lost because a resolver call failed. - **Not every update calls `save`.** Velt only sends a `save` when the PII actually changed and the action maps to a meaningful resolver event (add/update/delete of the comment body, etc.). Pure structural changes (status, priority, assignment) are handled by Velt and won't necessarily call your save. -- **Custom notifications only.** The notification resolver applies **only** to notifications whose source is `custom`. Standard comment/recording notifications are resolved through their own feature providers. See [Notifications](/self-host-data/notifications). +- **Custom notifications only.** The notification resolver applies **only** to notifications whose source is `custom`. Standard comment/recording notifications are resolved through their own feature providers. See [Notifications](/self-hosting/partial/notifications). - **Cross-organization notifications.** "For You" notifications can come from organizations other than the active one. Velt calls your notification/comment `get` endpoints with **each entry's own `organizationId`** — make sure your endpoints honor the `organizationId` in the request rather than assuming the current org. - **Activity is multi-feature.** A single activity record may contain PII owned by several providers. Velt resolves them in sequence (user → comment → reaction → recorder → activity), so register all relevant providers for activity feeds to render fully. diff --git a/self-host-data/reactions.mdx b/self-hosting/partial/reactions.mdx similarity index 98% rename from self-host-data/reactions.mdx rename to self-hosting/partial/reactions.mdx index 2762e53c5..5d522cc1f 100644 --- a/self-host-data/reactions.mdx +++ b/self-hosting/partial/reactions.mdx @@ -61,7 +61,7 @@ Both approaches are fully backward compatible and can be used together. Instead of implementing custom methods, you can configure endpoints directly and let the SDK handle HTTP requests. - `headers` may also be an async function resolved per request, and `credentials` (`'include' | 'same-origin' | 'omit'`) enables cookie/session auth — see [Async headers and credentials](/self-host-data/overview#async-headers-and-credentials) for details. + `headers` may also be an async function resolved per request, and `credentials` (`'include' | 'same-origin' | 'omit'`) enables cookie/session auth — see [Async headers and credentials](/self-hosting/partial/overview#async-headers-and-credentials) for details. ### getConfig @@ -565,10 +565,10 @@ Configuration for the reaction data provider. - `saveRetryConfig`: [`RetryConfig`](/api-reference/sdk/models/data-models#retryconfig). Configure retry behavior for save operations. - `deleteRetryConfig`: [`RetryConfig`](/api-reference/sdk/models/data-models#retryconfig). Configure retry behavior for delete operations. - `additionalFields`: `string[]`. Specify additional fields from the [`ReactionAnnotation`](/api-reference/sdk/models/data-models#reactionannotation) object to include in the resolver request payloads sent to your backend. By default, only core content fields are sent. Use this to request extra fields you need (e.g., `commentAnnotationId`, `type`). - - `fieldsToRemove`: `string[]`. Custom fields that are **moved** out of Velt's database into yours: they are sent to your backend, deleted from Velt's storage, and merged back on read via the resolver cache. Fields are matched on `!== undefined`, so falsy-but-meaningful values (`0`, `false`, `""`) are moved too. Use this for reaction-specific custom fields that must not be stored by Velt at all (e.g., `internalRef`, `tenantId`). See [Excluding & extending fields](/self-host-data/overview#excluding--extending-fields) for the full behavior comparison. + - `fieldsToRemove`: `string[]`. Custom fields that are **moved** out of Velt's database into yours: they are sent to your backend, deleted from Velt's storage, and merged back on read via the resolver cache. Fields are matched on `!== undefined`, so falsy-but-meaningful values (`0`, `false`, `""`) are moved too. Use this for reaction-specific custom fields that must not be stored by Velt at all (e.g., `internalRef`, `tenantId`). See [Excluding & extending fields](/self-hosting/partial/overview#excluding--extending-fields) for the full behavior comparison. -The reaction `icon` is always stripped from Velt's storage automatically — you don't need to list it in `fieldsToRemove`. Use `fieldsToRemove` for your own reaction-specific custom fields, and `additionalFields` for fields you want copied to your backend while keeping them in Velt. See the [Where supported](/self-host-data/overview#where-these-are-supported) matrix in the overview. +The reaction `icon` is always stripped from Velt's storage automatically — you don't need to list it in `fieldsToRemove`. Use `fieldsToRemove` for your own reaction-specific custom fields, and `additionalFields` for fields you want copied to your backend while keeping them in Velt. See the [Where supported](/self-hosting/partial/overview#where-these-are-supported) matrix in the overview. ```jsx diff --git a/self-host-data/recordings.mdx b/self-hosting/partial/recordings.mdx similarity index 99% rename from self-host-data/recordings.mdx rename to self-hosting/partial/recordings.mdx index 7e0dc7f12..20adf533a 100644 --- a/self-host-data/recordings.mdx +++ b/self-hosting/partial/recordings.mdx @@ -57,7 +57,7 @@ Both approaches are fully backward compatible and can be used together. Instead of implementing custom methods, you can configure endpoints directly and let the SDK handle HTTP requests. - `headers` may also be an async function resolved per request, and `credentials` (`'include' | 'same-origin' | 'omit'`) enables cookie/session auth — see [Async headers and credentials](/self-host-data/overview#async-headers-and-credentials) for details. + `headers` may also be an async function resolved per request, and `credentials` (`'include' | 'same-origin' | 'omit'`) enables cookie/session auth — see [Async headers and credentials](/self-hosting/partial/overview#async-headers-and-credentials) for details. ### getConfig @@ -402,7 +402,7 @@ Configuration for the recorder data provider. - `saveRetryConfig`: [`RetryConfig`](/api-reference/sdk/models/data-models#retryconfig). Configure retry behavior for save operations. - `deleteRetryConfig`: [`RetryConfig`](/api-reference/sdk/models/data-models#retryconfig). Configure retry behavior for delete operations. - `additionalFields`: `string[]`. Specify additional fields from the [`RecorderAnnotation`](/api-reference/sdk/models/data-models#recorderannotation) object to include in the resolver request payloads sent to your backend. By default, only core PII fields (identity, transcription, attachment URLs) are sent. Use this to request extra fields you need (e.g., `recordedTime`, `pageInfo`). - - `fieldsToRemove`: `string[]`. Custom fields that are **moved** out of Velt's database into yours: they are sent to your backend, deleted from Velt's storage, and merged back on read via the resolver cache. Fields are matched on `!== undefined`, so falsy-but-meaningful values (`0`, `false`, `""`) are moved too. Use this for recorder-specific custom fields that must not be stored by Velt at all (e.g., `internalRef`, `tenantId`). See [Excluding & extending fields](/self-host-data/overview#excluding--extending-fields) for the full behavior comparison. + - `fieldsToRemove`: `string[]`. Custom fields that are **moved** out of Velt's database into yours: they are sent to your backend, deleted from Velt's storage, and merged back on read via the resolver cache. Fields are matched on `!== undefined`, so falsy-but-meaningful values (`0`, `false`, `""`) are moved too. Use this for recorder-specific custom fields that must not be stored by Velt at all (e.g., `internalRef`, `tenantId`). See [Excluding & extending fields](/self-hosting/partial/overview#excluding--extending-fields) for the full behavior comparison. ```jsx const recorderResolverConfig = { diff --git a/self-host-data/users.mdx b/self-hosting/partial/users.mdx similarity index 99% rename from self-host-data/users.mdx rename to self-hosting/partial/users.mdx index 18726eea0..e687748ed 100644 --- a/self-host-data/users.mdx +++ b/self-hosting/partial/users.mdx @@ -39,7 +39,7 @@ Both approaches are fully backward compatible and can be used together. Instead of implementing custom methods, you can configure an endpoint directly and let the SDK handle HTTP requests. - `headers` may also be an async function resolved per request, and `credentials` (`'include' | 'same-origin' | 'omit'`) enables cookie/session auth — see [Async headers and credentials](/self-host-data/overview#async-headers-and-credentials) for details. + `headers` may also be an async function resolved per request, and `credentials` (`'include' | 'same-origin' | 'omit'`) enables cookie/session auth — see [Async headers and credentials](/self-hosting/partial/overview#async-headers-and-credentials) for details. ### getConfig