diff --git a/Dockerfile b/Dockerfile index 7c44a1c5a..63542d79a 100644 --- a/Dockerfile +++ b/Dockerfile @@ -63,6 +63,11 @@ ARG SUGABOTS_VERSION SUGABOTS_COMMIT ENV SUGABOTS_VERSION=$SUGABOTS_VERSION \ SUGABOTS_COMMIT=$SUGABOTS_COMMIT +# Where the deployment guide and compose.yml mount the files volume, for +# BLOB_STORE_FILESYSTEM_ROOT. Owned by the app's user so a new volume mounted +# on it starts out writable. +RUN mkdir -p /data/files && chown node:node /data/files + USER node EXPOSE 3000 diff --git a/packages/docs/content/configuration.mdx b/packages/docs/content/configuration.mdx index 93577df96..889c95027 100644 --- a/packages/docs/content/configuration.mdx +++ b/packages/docs/content/configuration.mdx @@ -61,11 +61,11 @@ Generate each with `openssl rand -base64 32`. Keep them private and backed up. ## File storage -| Variable | Default | Description | -| ---------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `BLOB_STORE_PROVIDER` | `postgres` | Where files are kept, such as tool results too large to hand a model whole. `postgres` keeps them in the database, up to 100 MiB each. `filesystem` keeps them on disk, for a single server. | -| `BLOB_STORE_FILESYSTEM_ROOT` | Required for `filesystem` | The directory files are kept under, such as a mounted volume. Back it up with the database. | -| `THREAD_FILE_MAX_BYTES` | The blob store's limit | The largest file a thread keeps, in bytes, when lower than what the blob store keeps. | +| Variable | Default | Description | +| ---------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `BLOB_STORE_PROVIDER` | `postgres` | Where files are kept, such as tool results too large to hand a model whole. `postgres` keeps them in the database, up to 100 MiB each. `filesystem` keeps them on disk, for a single server. Choose before any files are kept: changing it doesn't move them. | +| `BLOB_STORE_FILESYSTEM_ROOT` | Required for `filesystem` | The directory files are kept under. In the Docker image, mount a volume at `/data/files`. A [folder from the server](/docs/deploy-with-docker#files-on-a-volume) must be writable by uid 1000. Back it up with the database. | +| `THREAD_FILE_MAX_BYTES` | The blob store's limit | The largest file a thread keeps, in bytes, when lower than what the blob store keeps. | ## Web app diff --git a/packages/docs/content/deploy-with-docker.mdx b/packages/docs/content/deploy-with-docker.mdx index 660811a6d..a4b585629 100644 --- a/packages/docs/content/deploy-with-docker.mdx +++ b/packages/docs/content/deploy-with-docker.mdx @@ -70,6 +70,9 @@ both the web app and the API. It stores everything in PostgreSQL 18. emails and sign-in redirects are built from it. Every other setting is in the [configuration reference](/docs/configuration). + To keep files on a volume rather than in the database, add + [those settings](#files-on-a-volume) now, before anyone uses Sugabots. + 2. Create `.env` In the same folder, generate the database password and the two secret keys: @@ -189,6 +192,61 @@ Either way, the service needs these settings: See the [configuration reference](/docs/configuration) for what each does. +## Files on a volume + +Sugabots keeps files, such as attachments and large tool results, in the +database unless told otherwise. There, every file adds to the database and its +backups, and each one is held in memory while it's saved. To keep them on a +volume instead, add these settings and a `files` volume to `compose.yml`, +keeping what's already there: + +```yaml title="compose.yml" +services: + sugabots: + environment: + BLOB_STORE_PROVIDER: filesystem + BLOB_STORE_FILESYSTEM_ROOT: /data/files + volumes: + - files:/data/files + +volumes: + postgres_data: + files: +``` + +Add both: without the volume, files are lost when the container is replaced, +such as when you update. Then run `docker compose up -d`. The `compose.yml` +the [Quickstart](/docs/quickstart) downloads has these lines, commented out. + +Choose before you start using Sugabots. Files already kept stay in the +database, and Sugabots can't open them unless you switch back. + +To keep files in a folder on the server rather than a Docker volume, make the +folder and give it to the user Sugabots runs as: + +```sh +sudo mkdir -p /srv/sugabots/files +sudo chown 1000:1000 /srv/sugabots/files +``` + +Keep the two settings, but mount the folder in place of `files:/data/files`, +and leave out `files:` at the end: + +```yaml title="compose.yml" +services: + sugabots: + volumes: + - /srv/sugabots/files:/data/files +``` + +If Sugabots can't write to the folder, it stops, and +`docker compose logs sugabots` says why. + +On a hosting platform, attach a persistent disk at `/data/files`, set the same +two settings, and run one copy of the service: the disk belongs to one server. +Without a disk, keep the default: files kept in the container are lost when +it's replaced. + ## Email Sugabots sends email for invites and address verification. Without a provider, @@ -299,8 +357,26 @@ This deletes whatever is in the database before loading the backup. If the backup is from an older version, Sugabots applies the newer migrations when it starts. -On a hosting platform, use the platform's database backups, and keep a copy of -the two keys. +If you keep [files on a volume](#files-on-a-volume), back it up after the +database, and restore it from the same backup: the database lists the files, +and the volume holds them. In the same folder, write the volume to a file: + +```sh +docker run --rm -v sugabots_files:/data/files -v "$PWD":/backup alpine \ + tar czf /backup/sugabots-files.tar.gz -C /data/files . +``` + +To restore it, run this before `docker compose start sugabots` above: + +```sh +docker run --rm -v sugabots_files:/data/files -v "$PWD":/backup alpine \ + tar xzf /backup/sugabots-files.tar.gz -C /data/files +``` + +For a folder on the server, archive the folder itself instead. + +On a hosting platform, use the platform's database backups, and its disk +backups if you keep files on a disk. Keep a copy of the two keys too. ## Building the image yourself diff --git a/packages/website/public/compose.yml b/packages/website/public/compose.yml index e91ed45f7..2d175a687 100644 --- a/packages/website/public/compose.yml +++ b/packages/website/public/compose.yml @@ -15,6 +15,14 @@ services: BETTER_AUTH_SECRET: ${BETTER_AUTH_SECRET:?Set it in .env} CREDENTIALS_ENCRYPTION_KEY: ${CREDENTIALS_ENCRYPTION_KEY:?Set it in .env} EMAIL_TRANSACTIONAL_FROM: Sugabots + # Files are kept in the database. To keep them on a volume instead, do it + # before you use Sugabots: uncomment the two settings, the two volumes + # lines below and `files:` at the end, then run `docker compose up -d`. + # https://sugabots.ai/docs/deploy-with-docker#files-on-a-volume + # BLOB_STORE_PROVIDER: filesystem + # BLOB_STORE_FILESYSTEM_ROOT: /data/files + # volumes: + # - files:/data/files depends_on: postgres: condition: service_healthy @@ -36,3 +44,4 @@ services: volumes: postgres_data: + # files: