diff --git a/README.md b/README.md index 77ae7ae..5ae43c6 100644 --- a/README.md +++ b/README.md @@ -150,6 +150,11 @@ it: GOCHAT_URL=http://localhost:8080 GOCHAT_BOT_TOKEN=b_... go run ./examples/echobot -state echobot.pts ``` +[docs/bots.md](docs/bots.md) is the guide to building bots: making one, +the three ways to get updates, commands and buttons, limits, and running +one in production. Its examples include a Python bot that long-polls with +nothing but the standard library, and a webhook bot. + ### Client API for native apps The [mobile apps](mobile) speak [the GoChat protocol](docs/protocol.md) @@ -835,8 +840,8 @@ api/protobuf/ protobuf sources and generated protobuf + DRPC code cmd/ cobra commands and embedded default configs config/ configs for host runs and for the Docker stack deployments/ Dockerfile and compose files -docs/ the GoChat protocol's specification -examples/ bots built on pkg/gochat +docs/ the GoChat protocol's specification and the bots guide +examples/ bots: over the socket, long polling and a webhook internal/actors ask, pools and middleware on top of hollywood internal/app wiring of the server and web processes internal/authz casbin model and enforcer diff --git a/docs/bots.md b/docs/bots.md new file mode 100644 index 0000000..d8b2441 --- /dev/null +++ b/docs/bots.md @@ -0,0 +1,486 @@ +# Building GoChat bots + +A bot is a GoChat account that a program signs in to with a token. It +receives the messages meant for it as updates and answers with the same calls +the apps make. This guide takes a bot from nothing to production; +[protocol.md](protocol.md#bots) is the specification it leans on. + +- [Making a bot](#making-a-bot) +- [A first call](#a-first-call) +- [Getting updates](#getting-updates) +- [Commands](#commands) +- [Buttons](#buttons) +- [Writing to a person first](#writing-to-a-person-first) +- [What a bot sees](#what-a-bot-sees) +- [Errors](#errors) +- [Limits](#limits) +- [What a bot may not call](#what-a-bot-may-not-call) +- [Running a bot](#running-a-bot) +- [The examples](#the-examples) + +## Making a bot + +1. On the web, open **Settings → Bots**. Under **New bot**, give it a + username ending in `bot`, such as `weather_bot`, and a display name, and + press **Create bot**. +2. The page shows the bot's token, `b_` and 43 characters, once. Copy it. + The server keeps only its hash: a lost token cannot be shown again, only + reset. +3. Under **Manage**, pick a team in **Add to** and press **Add**. You may add + a bot to the teams you may invite people to. +4. The bot is in no channel yet. In each channel it should work in, open + **Members → Add people** and find the bot. Anyone may add it to a public + channel; a private channel's admins only. The bot may also join a public + channel of its teams itself, with `ChannelService/JoinChannel`. People in + its teams may write to it directly, and it may + [write to them first](#writing-to-a-person-first). + +Whoever holds the token acts as the bot. Keep it out of code and +repositories; hand it to the program in its environment or a secret store, +and never log it. **Manage → Reset token** makes a new one and stops the old +one at once: the bot's sockets close and its HTTP calls answer 401. +**Delete** removes the bot; its messages stay. + +Programs may do the same with `BotService`: `CreateBot`, `AddBotToTeam`, +`UpdateBot`, `ResetBotToken` and `DeleteBot`, called by the owner. + +## A first call + +The HTTP Bot API is the quickest way to try a token: + +```sh +export GOCHAT_URL=https://chat.example.com +export GOCHAT_BOT_TOKEN=b_... + +curl -sS -X POST "$GOCHAT_URL/api/v2/bot/getMe" \ + -H "Authorization: Bearer $GOCHAT_BOT_TOKEN" +``` + +```json +{"ok":true,"result":{"user":{"id":"bc3bd805-...","username":"weather_bot","display_name":"Weather","bot":true,"bot_owner_id":"8976d5b8-...",...},"session_id":"9703494b..."}} +``` + +To post, the bot needs a channel's id. The bot's `channel_joined` update +carries it when someone adds the bot, `listChannels {"team_id": "..."}` +lists the channels the bot is in, and the channel's address on the web ends +in it: `/t//ch/`. + +```sh +curl -sS -X POST "$GOCHAT_URL/api/v2/bot/sendMessage" \ + -H "Authorization: Bearer $GOCHAT_BOT_TOKEN" -H "Content-Type: application/json" \ + -d '{"channel_id": "b9cf4aed-...", "content": "Hello from **curl**"}' +``` + +The answer is `{"ok": true, "result": {"message": {...}}}`, the message as +posted. Every method takes its call's request as JSON with the proto field +names (`channel_id`, `root_id`), and the table in +[protocol.md](protocol.md#the-http-bot-api) maps the short names to calls. + +## Getting updates + +A bot gets what happens to it in one numbered sequence, its account's: each +update carries `pts`, and the bot keeps the last one it handled. There are +three ways to take them, and a bot uses one: + +| | Socket | Long polling | Webhook | +| --- | --- | --- | --- | +| How | `pkg/gochat`, or any client of the protocol | `getUpdates` in a loop | the server POSTs each update | +| The bot needs | a process that stays up | a process that stays up | an https address the server reaches | +| Who keeps `pts` | the bot | the server | the server | + +Pick the socket for Go, or for any bot that keeps a process running and +wants updates the moment they happen. Pick long polling for other languages, +or a bot behind a firewall: it needs nothing but HTTPS and JSON. Pick a +webhook when the bot runs where it is called rather than where it waits, +behind a load balancer or as a function. + +### The socket, with pkg/gochat + +[`pkg/gochat`](../pkg/gochat) connects, reconnects with backoff, and resumes +after the last update it handed out: + +```go +c, err := gochat.Dial(ctx, gochat.Options{URL: url, Token: token, Client: "weatherbot/1", Pts: saved}) +if err != nil { + return err +} +defer c.Close() +for u := range c.Updates() { + if m := u.Event.GetMessagePosted().GetMessage(); m != nil { + // ... + } + save(c.Pts()) +} +return c.Err() +``` + +- `c.Call(ctx, "gochat.v1.MessageService/PostMessage", req, res)` makes a + call. A failure the server reports is a `*gochat.Error` with the code of + [protocol.md](protocol.md#calls). `gochat.ErrLost` means the connection + dropped before the answer came: the server may have made the call, so the + client does not make it again. +- `u.Reload` is set when the bot missed more than the server replays, or was + away longer than it keeps updates (a week by default): what came meanwhile + is gone. `u.Ephemeral` marks updates without a number, such as typing. +- `Updates` closes when the client stops for good: the token is not valid + (4001), the client broke the protocol (4002), or the server takes its + layer no more (4011). `c.Err()` says why. + +[examples/echobot](../examples/echobot) is a whole bot: + +```sh +GOCHAT_URL=https://chat.example.com GOCHAT_BOT_TOKEN=b_... \ + go run ./examples/echobot -state echobot.pts +``` + +Bots in other languages speak the protocol as +[protocol.md](protocol.md#connecting) says, with code generated from +[`protocol.proto`](../api/protobuf/proto/gochat/v1/protocol.proto). + +### Long polling with getUpdates + +```sh +curl -sS -X POST "$GOCHAT_URL/api/v2/bot/getUpdates" \ + -H "Authorization: Bearer $GOCHAT_BOT_TOKEN" -H "Content-Type: application/json" \ + -d '{"offset": 0, "timeout": 50}' +``` + +```json +{"ok":true,"result":{"updates":[ + {"team_joined":{"team":{...}},"pts":"1","pts_count":1}, + {"channel_joined":{"channel":{"id":"b9cf4aed-...","name":"town-square",...}},"pts":"2","pts_count":1} +],"pts":"2"}} +``` + +- Each update is an `Event` of + [`gateway.proto`](../api/protobuf/proto/gochat/v1/gateway.proto) in + protobuf's JSON mapping: one field names what happened (`message_posted`, + `callback_query`, `channel_joined`, ...). 64-bit numbers such as `pts` + are strings, enums are names, and fields left at zero are left out, an + empty `updates` too. Skip fields you do not know: later versions add + some. +- `offset` is the last `pts` handled plus one. It confirms the updates + before it, and the server remembers: `0`, or no offset, starts after the + last update confirmed, after a restart too. An update handled but not yet + confirmed, because the bot stopped before its next call, comes again. +- `timeout` is how long the call waits for an update when none is waiting, + at most 50 seconds; give the HTTP client longer. `limit` is at most 100. +- `too_long` is set when the bot was away longer than the server keeps + updates. The result holds none: take `pts` plus one as the next offset. +- A bot polls once at a time. A second `getUpdates` ends the one waiting + with 409, so two copies of a bot end each other's polls over and over. + While a webhook is set, `getUpdates` answers 409 too. +- `getWebhookInfo` tells `delivered_pts`, the last update confirmed, and + `pending`, how many wait, with a webhook or without. + +[examples/dicebot](../examples/dicebot) is a whole bot in Python with +nothing but the standard library: + +```sh +GOCHAT_URL=https://chat.example.com GOCHAT_BOT_TOKEN=b_... \ + python3 examples/dicebot/dicebot.py +``` + +### Webhooks + +```sh +export GOCHAT_WEBHOOK_SECRET=$(openssl rand -hex 32) + +curl -sS -X POST "$GOCHAT_URL/api/v2/bot/setWebhook" \ + -H "Authorization: Bearer $GOCHAT_BOT_TOKEN" -H "Content-Type: application/json" \ + -d '{"url": "https://bot.example.com/gochat", "secret": "'"$GOCHAT_WEBHOOK_SECRET"'"}' +``` + +The server then POSTs each update to the URL, as JSON like one of +`getUpdates`' updates, with the secret in `X-GoChat-Bot-Secret`. + +- The URL is https, without credentials, and reaches a public address. The + server refuses loopback, private, link-local and other internal + addresses, a name that resolves to one at the time of each delivery too, + and follows no redirect. For development only, a server with + `services.bots.allow_private_webhooks` set takes plain http and local + addresses. +- The secret is up to 256 letters, digits, `_` and `-`. Always set one, and + answer 401 to a request without it, comparing in constant time: the URL + alone does not prove a request came from the server. +- A 2xx answer takes the update. Anything else, or no answer within 10 + seconds, and the server sends it again, after 1 second, then 2, 4, and so + on up to 5 minutes, until it is taken. +- Updates go one at a time, in order: the next waits until this one is + taken. Answer quickly, and answer 2xx to updates the bot ignores or + cannot handle, or they hold back every update after them. A delivery that + timed out may have been handled: expect one twice now and then. +- A new webhook gets the updates after it was set. Changing its URL or its + secret keeps its place, and `deleteWebhook` hands the place to + `getUpdates`. +- When updates stop coming, `getWebhookInfo` tells the last error, such as + `the webhook answered 401`, and when it happened. + +[examples/webhookbot](../examples/webhookbot) takes deliveries and answers +over the HTTP Bot API: + +```sh +GOCHAT_URL=https://chat.example.com GOCHAT_BOT_TOKEN=b_... GOCHAT_WEBHOOK_SECRET=... \ + go run ./examples/webhookbot -addr :8443 -cert cert.pem -key key.pem +``` + +Without `-cert` it serves plain http, for a TLS-terminating proxy in front. + +## Commands + +`setCommands` replaces the list of the bot's commands, which people see +suggested when they type `/` in a channel with the bot: + +```json +{"commands": [{"name": "roll", "description": "Roll a die: /roll or /roll 20"}]} +``` + +A bot offers up to 100. A name is 1–32 lowercase letters, digits and +underscores, and a description 1–256 characters. + +A command is an ordinary message, `/roll 20`, that everyone in the channel +sees. It reaches every bot in the channel, those in privacy mode too, so +with several bots people address one: `/roll@dice_bot 20`. A bot ignores +commands addressed to another bot. `gochat.Command(text, me.Username)` +reads a command this way, and the Python example has its own `command()`. +The arguments are what follows the first space. + +To answer in the thread the command was posted in, pass its `root_id` back +in `sendMessage`. + +## Buttons + +`markup` in `sendMessage` or `editMessage` puts buttons under a message: + +```sh +curl -sS -X POST "$GOCHAT_URL/api/v2/bot/sendMessage" \ + -H "Authorization: Bearer $GOCHAT_BOT_TOKEN" -H "Content-Type: application/json" \ + -d '{ + "channel_id": "b9cf4aed-...", + "content": "Deploy v1.4 to production?", + "markup": {"rows": [{"buttons": [ + {"text": "Deploy", "callback_data": "deploy:v1.4"}, + {"text": "Changelog", "url": "https://example.com/changes/v1.4"} + ]}]} + }' +``` + +A button opens its `url`, or sends the bot a `callback_query` update with +its `callback_data`, the message and the person who pressed it, `from`: + +```json +{"callback_query":{"id":"d23781af-...","from":{"id":"8976d5b8-...","username":"ada",...},"message":{"id":"4b035592-...",...},"data":"deploy:v1.4"},"pts":"9","pts_count":1} +``` + +- Answer every press within a minute with `answerCallback`: + `{"query_id": "d23781af-...", "text": "Deploying."}`. The presser sees the + text as a passing notice, or as an alert with `"alert": true`; an empty + text shows nothing. A press answered or older than a minute answers 404. +- The server lets people press only the buttons a message has, so `data` + is always one the bot put there. Who pressed it is up to the bot to + check, before it deploys anything. +- `callback_data` holds 64 bytes: keep the state in the bot and a key in the + data. +- `editMessage` replaces a message's buttons: send them again to keep them, + and leave `markup` out to remove them. A reply's edit takes its `root_id`. + +## Writing to a person first + +For reminders, alerts and notices, `openDirect` opens the bot's direct +conversation with a person of one of its teams, and the bot posts there as +in any channel: + +```sh +curl -sS -X POST "$GOCHAT_URL/api/v2/bot/openDirect" \ + -H "Authorization: Bearer $GOCHAT_BOT_TOKEN" -H "Content-Type: application/json" \ + -d '{"team_id": "e04f3c5f-...", "user_id": "8976d5b8-..."}' +``` + +```json +{"ok":true,"result":{"channel":{"id":"f50011f8-...","type":"CHANNEL_TYPE_DIRECT",...}}} +``` + +Over the socket, `c.OpenDirect(ctx, teamID, userID)` does the same. The +team is often that of a message the person posted, its `message_posted`'s +`team_id`, and the person its author: echobot's `/remindme 10m stand up` +writes back that way. + +- The person is an active member of the team, as the bot is, or the call + answers 404. A person who blocked the bot refuses it with 403, and the + bot's posts in a conversation they have with it fail the same way. +- A bot opens no group conversation and none with another bot. +- A bot opens up to 30 new conversations an hour; past that the call + answers 429. Opening one that exists already, whoever opened it, costs + nothing and returns it. +- The person sees it as any direct conversation, with the bot's badge, and + may mute it, block the bot, or delete it for themselves. The bot receives + every message of it. + +Write to people who asked for it, or who expect it: a bot that writes +uninvited gets blocked. + +## What a bot sees + +A bot starts in **privacy mode**. From its channels it receives: + +- messages that start with `/`, and those that mention it; +- replies in the threads it posted in; +- every message of its direct conversations; +- edits, deletions and reactions of those messages, and changes to its + channels; +- `callback_query` when its buttons are pressed; +- `team_joined`, `channel_joined` and `channel_left` as it is added and + removed. + +It never receives the posts of its own messages. It reads only what it was +sent: `getMessage` of another message answers 404, and `ListMessages` and +`ListThread` answer 403. + +With **Gets every message** checked under **Manage** (`sees_all` in +`UpdateBot`), a bot receives every message of the channels it is added to +from then on, and may read their history. A channel keeps the mode the bot +had when it was added: to change it, remove the bot from the channel and +add it again. A team's `town-square`, which every member stays in, cannot +be left, so choose before adding a bot there. Ask for `sees_all` only when +the bot needs it: it then reads everything people say. + +## Errors + +The HTTP Bot API answers an error as + +```json +{"ok": false, "error_code": 403, "description": "join the channel first"} +``` + +with `error_code` the HTTP status. Over the socket the same failures carry +the protocol's codes. + +| Status | Code | Meaning | What to do | +| --- | --- | --- | --- | +| 400 | 3 | The request is not valid: a field, a size, JSON that does not parse. | Fix the request. | +| 401 | 16 | The token is not valid: reset, or the bot deleted. | Stop. | +| 403 | 7 | Not allowed: not in the channel, someone else's message, history in privacy mode. | Do not retry. | +| 404 | 5, 12 | No such thing, or a method that does not exist or is not open to bots. | Do not retry. | +| 409 | 6, 9 | It exists, or not now: a second `getUpdates`, a webhook set, a channel that cannot be left. | Depends on the call. | +| 429 | 8 | Too many requests, or a limit reached. | Wait, then retry. | +| 500 | 13 | The server failed. | Retry later. | +| 503 | 14 | Unavailable. | Retry with backoff. | + +Calls over the [request budget](#limits) are refused before they reach the +Bot API: 429 with `Retry-After`, and an `ApiError` as the body, +`{"code": 8, "message": "too many requests, slow down"}`, rather than +`ok` and `description`. + +Retry 429, 5xx and network failures with backoff; other errors come again +the same way. A call whose answer never came may have been made: posting +it again may post twice. + +## Limits + +These are the defaults; a server's operator may change them. + +| What | Limit | +| --- | --- | +| The request budget | 30 calls a second, in bursts of up to 120, per bot, over HTTP and the socket together (`web.rate_limit`); ten times that per address | +| Posting | 5 messages a second in a channel, in bursts of 20 (`services.channels.post_rate`, `post_burst`); more answers 429 `you are posting too fast` | +| A socket's calls | 32 in flight and 256 waiting their turn | +| A socket's frames | 10 a second, in bursts of 40, each up to 64 KiB (`gateway.frame_rate`, `frame_burst`); faster, and the server closes the socket and fails the calls in flight | +| Sockets | 20 per account (`gateway.max_per_user`); more closes with 4008 | +| An HTTP request | 1 MiB | +| A message | 16000 characters (`services.channels.max_message_length`) | +| Buttons | 8 rows of 8; a text of 64 characters; `callback_data` of 64 bytes | +| A press's answer | 200 characters, within a minute | +| Commands | 100; names of 1–32 characters, descriptions of 1–256 | +| `getUpdates` | waits at most 50 seconds; at most 100 updates | +| A webhook | 10 seconds to answer (`services.bots.webhook_timeout`); retries up to 5 minutes apart (`webhook_retry_max`) | +| Updates kept | a week (`services.updates.retention`) | +| New direct conversations | 30 an hour per bot (`services.directory.max_bot_conversations_per_hour`) | +| Bots | 20 per owner (`services.bots.max_per_owner`) | + +A bot that makes calls in bursts over the socket paces them itself: +`pkg/gochat` sends each call as it is made. + +## What a bot may not call + +Bots make only the calls of +[the table in protocol.md](protocol.md#what-bots-may-call); any other +answers 404, `bots cannot make this call`, over HTTP and code 12 over the +socket. Among those it cannot make: creating channels, teams or group +conversations, adding people to them, inviting, pinning, saving, searching, +pressing buttons, managing bots, and anything of `AdminService` or +`SafetyService`. It deletes and removes link previews of its own messages +only, and `ListMessages` and `ListThread` need `sees_all`. + +## Running a bot + +**Run one copy.** Two copies of a socket bot both get every update and both +answer; two polling ones end each other's polls. Several receivers may share +a webhook's URL, but the server delivers one update at a time anyway. + +**Use a service manager or a container** that restarts the bot when it +fails. A systemd unit, with the token in a file only root reads: + +```ini +# /etc/systemd/system/dicebot.service +[Unit] +Description=Dice bot for GoChat +Wants=network-online.target +After=network-online.target + +[Service] +ExecStart=/usr/bin/python3 -u /opt/dicebot/dicebot.py +# GOCHAT_URL=... and GOCHAT_BOT_TOKEN=..., mode 0600 +EnvironmentFile=/etc/dicebot.env +DynamicUser=yes +Restart=on-failure +RestartSec=5s + +[Install] +WantedBy=multi-user.target +``` + +A Go bot that keeps its `pts` adds `StateDirectory=echobot` and +`ExecStart=/usr/local/bin/echobot -state /var/lib/echobot/pts`. In a +container: + +```dockerfile +FROM python:3.13-alpine +COPY examples/dicebot/dicebot.py /app/dicebot.py +USER nobody +CMD ["python3", "-u", "/app/dicebot.py"] +``` + +```sh +docker build -t dicebot -f dicebot.Dockerfile . +docker run -d --name dicebot --restart unless-stopped --env-file dicebot.env dicebot +``` + +A bot whose token was reset exits, and starting it again fails the same +way until it has the new token. + +**Reconnect with backoff and jitter.** `pkg/gochat` does it: it waits up +to half a second at first, twice as long after each failure up to 30 +seconds, and starts over after a connection that lasted a minute. The +server also closes a bot's socket every day, when its session renews, and +the client reconnects. A client of the protocol of your own follows the +close codes of [protocol.md](protocol.md#connecting). A polling bot backs +off on network failures, 5xx and 409, waits out `Retry-After` on 429, and +stops on 401. A webhook's retries are the server's: the receiver needs only +to come back. + +**Keep the last `pts`.** With `getUpdates` or a webhook the server keeps it. +Over the socket the bot does: after handling each update with a number, +store `c.Pts()` in a file or a database, and pass it back in +`Options.Pts`. Write the file whole, to a new file renamed over the old, +so a crash never leaves half a number. Started with `0`, the bot gets only +what comes from then on, and misses what came while it was down. Stored +after handling, an update handled just before a crash comes again; stored +before, it is lost: choose by what the bot does twice more safely. + +## The examples + +| | Updates | Language | Shows | +| --- | --- | --- | --- | +| [echobot](../examples/echobot) | the socket | Go, `pkg/gochat` | commands, a button and its answer, writing to a person first, keeping `pts` | +| [dicebot](../examples/dicebot) | `getUpdates` | Python 3, standard library | commands and `@bot` addressing, a button that edits its message | +| [webhookbot](../examples/webhookbot) | a webhook | Go | checking the secret, answering 2xx, calling the HTTP Bot API | diff --git a/docs/protocol.md b/docs/protocol.md index 3786778..53d6ce8 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -101,8 +101,11 @@ or an `ApiError`: | 16 | Unauthenticated | Ids are the client's to choose, unique among its calls in flight. Results -come in any order. A socket has at most 32 calls in flight; more wait -their turn. Calls spend the session's request budget, as the HTTP API's do. +come in any order. A socket has at most 32 calls in flight, and up to 256 +more wait their turn; past that the server closes it with 4009. A client +sends at most 10 frames a second, in bursts of 40: faster, and the server +closes the socket with 1008. Calls spend the session's request budget, as +the HTTP API's do. People may make the unary calls the apps make over `/api/v1` (see [`internal/web/api.go`](../internal/web/api.go)), plus `UpdatesService` and @@ -229,6 +232,8 @@ message. A bot opts its own messages out with `no_link_preview`. ## Bots +[bots.md](bots.md) walks through building one, with examples. + A bot is an account a person owns. People make bots in the web's settings, or with `BotService.CreateBot`, which answers the bot and its token once: @@ -259,9 +264,14 @@ By default a bot is in **privacy mode** and receives, from its channels: - `CallbackQuery` when someone presses a button of its messages; - `ChannelJoined` and `ChannelLeft` as it is added to channels and removed. +It also receives `TeamJoined` and `TeamLeft` as it joins and leaves teams. +In privacy mode it reads only what it was sent: `GetMessage` of another +message answers code 5, and `ListMessages` and `ListThread` code 7. + With `sees_all` on (`UpdateBot`), a bot added to a channel from then on receives every message there and may read its history. A channel keeps -what the bot had when it was added; add it again to change it. +what the bot had when it was added; add it again to change it. No one +leaves a team's `town-square`, so a bot keeps its mode there for good. ### What bots may call @@ -271,7 +281,7 @@ what the bot had when it was added; add it again to change it. | `UserService` | `GetUser`, `UpdateProfile` | | `TeamService` | `ListTeams`, `GetTeam`, `ListTeamMembers` | | `ChannelService` | `ListChannels`, `GetChannel`, `ListChannelMembers`, `JoinChannel`, `LeaveChannel`, `OpenDirectChannel` | -| `MessageService` | `PostMessage`, `EditMessage`, `DeleteMessage` (its own), `GetMessage`, `ListMessages`, `ListThread` (with `sees_all`), `SendTyping`, `React`, `RemoveLinkPreview` (its own) | +| `MessageService` | `PostMessage`, `EditMessage`, `DeleteMessage` (its own), `GetMessage`, `ListMessages` and `ListThread` (with `sees_all`), `SendTyping`, `React`, `RemoveLinkPreview` (its own) | | `UpdatesService` | `GetState`, `GetDifference` | | `BotService` | `SetCommands`, `GetCommands`, `AnswerCallback`, `SetWebhook`, `DeleteWebhook`, `GetWebhookInfo`, `ConfirmUpdates` | | `FileService` | uploads and downloads, at `/api/v2/files` | @@ -338,9 +348,13 @@ proto field names), and the answer is always JSON: ```json {"ok": true, "result": { ... }} -{"ok": false, "error_code": 7, "description": "permission denied"} +{"ok": false, "error_code": 403, "description": "permission denied"} ``` +`error_code` is the answer's HTTP status. Calls over the request budget are +refused before they reach the Bot API: 429 with `Retry-After` and an +`ApiError`, `{"code": 8, "message": "too many requests, slow down"}`. + | Method | Call | | --- | --- | | `getMe` | `AuthService/Me` | diff --git a/examples/dicebot/dicebot.py b/examples/dicebot/dicebot.py new file mode 100755 index 0000000..e41db8b --- /dev/null +++ b/examples/dicebot/dicebot.py @@ -0,0 +1,143 @@ +#!/usr/bin/env python3 +"""A GoChat bot over the HTTP Bot API, with long polling and nothing but +Python's standard library: /roll answers a die's roll under a Roll again +button, and /roll 20 rolls a twenty-sided one. + + GOCHAT_URL=https://chat.example.com GOCHAT_BOT_TOKEN=b_... python3 examples/dicebot/dicebot.py + +The server remembers the last update the bot confirmed, so after a restart +the bot gets what it missed. docs/bots.md walks through it. +""" + +import json +import os +import random +import sys +import time +import urllib.error +import urllib.request + +URL = os.environ.get("GOCHAT_URL", "").rstrip("/") +TOKEN = os.environ.get("GOCHAT_BOT_TOKEN", "") +# How long getUpdates waits for an update, at most 50 seconds. +POLL = 50 + + +class APIError(Exception): + def __init__(self, status, description, retry_after=0): + super().__init__(f"{description} ({status})") + self.status = status + self.retry_after = retry_after + + +def call(method, body=None, timeout=10): + """Makes one call of the HTTP Bot API and answers its result.""" + req = urllib.request.Request( + f"{URL}/api/v2/bot/{method}", + data=json.dumps(body or {}).encode(), + headers={"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}, + ) + try: + with urllib.request.urlopen(req, timeout=timeout) as res: + return json.load(res).get("result", {}) + except urllib.error.HTTPError as e: + try: + answer = json.load(e) + except ValueError: + answer = {} + # Calls over the request budget answer 429 with an ApiError and + # Retry-After. + description = answer.get("description") or answer.get("message") or e.reason + raise APIError(e.code, description, int(e.headers.get("Retry-After") or 0)) from None + + +def command(text, bot): + """Reads "/roll 20", or "/roll@dice_bot 20" addressed to this bot.""" + if not text.startswith("/"): + return None + word, _, args = text[1:].partition(" ") + name, at, to = word.partition("@") + if not name or (at and to.lower() != bot.lower()): + return None + return name.lower(), args.strip() + + +def roll(sides): + """A roll and its button, which carries the die in its data.""" + text = f"🎲 d{sides}: {random.randint(1, sides)}" + markup = {"rows": [{"buttons": [{"text": "Roll again", "callback_data": f"roll:{sides}"}]}]} + return text, markup + + +def handle(update, me): + m = update.get("message_posted", {}).get("message") + if m and m.get("author", {}).get("id") != me["id"]: + cmd = command(m.get("content", ""), me["username"]) + if cmd and cmd[0] == "roll": + sides = int(cmd[1]) if cmd[1].isdigit() and 2 <= int(cmd[1]) <= 1000 else 6 + text, markup = roll(sides) + call("sendMessage", { + "channel_id": m["channel_id"], "root_id": m.get("root_id", ""), + "content": text, "markup": markup, + }) + return + q = update.get("callback_query") + if q: + kind, _, sides = q.get("data", "").partition(":") + if kind != "roll" or not sides.isdigit(): + call("answerCallback", {"query_id": q["id"]}) + return + text, markup = roll(int(sides)) + msg = q["message"] + # Editing replaces the buttons: send them again to keep them. + call("editMessage", { + "channel_id": msg["channel_id"], "message_id": msg["id"], "root_id": msg.get("root_id", ""), + "content": text, "markup": markup, + }) + call("answerCallback", {"query_id": q["id"], "text": "Rolled again."}) + + +def main(): + if not URL or not TOKEN: + sys.exit("set GOCHAT_URL and GOCHAT_BOT_TOKEN") + me = call("getMe")["user"] + print(f"signed in as @{me['username']}", flush=True) + call("setCommands", {"commands": [{"name": "roll", "description": "Roll a die: /roll or /roll 20"}]}) + + # 0 starts after the last update the server knows the bot handled. + offset, delay = 0, 1 + while True: + try: + res = call("getUpdates", {"offset": offset, "timeout": POLL}, timeout=POLL + 10) + except APIError as e: + if e.status == 401: + sys.exit("the token is not valid: was it reset?") + # 409: another copy of the bot is polling, or a webhook is set. + print(f"getUpdates: {e}", file=sys.stderr, flush=True) + time.sleep(e.retry_after or delay) + delay = min(delay * 2, 60) + continue + except OSError as e: + print(f"getUpdates: {e}", file=sys.stderr, flush=True) + time.sleep(delay) + delay = min(delay * 2, 60) + continue + delay = 1 + for u in res.get("updates", []): + # One update the bot cannot handle must not hold back the rest. + try: + handle(u, me) + except Exception as e: + print(f"update {u['pts']}: {e!r}", file=sys.stderr, flush=True) + # Numbers are strings in JSON: int64 in protobuf's mapping. + offset = int(u["pts"]) + 1 + if res.get("too_long"): + print("updates were lost while the bot was away", file=sys.stderr, flush=True) + offset = int(res["pts"]) + 1 + + +if __name__ == "__main__": + try: + main() + except KeyboardInterrupt: + pass diff --git a/examples/echobot/main.go b/examples/echobot/main.go index b531621..ec7f141 100644 --- a/examples/echobot/main.go +++ b/examples/echobot/main.go @@ -156,7 +156,13 @@ func save(path string, pts int64) { if path == "" { return } - if err := os.WriteFile(path, []byte(strconv.FormatInt(pts, 10)), 0o600); err != nil { + // Renamed over the old, the file never holds half a number. + tmp := path + ".tmp" + if err := os.WriteFile(tmp, []byte(strconv.FormatInt(pts, 10)), 0o600); err != nil { + log.Printf("saving the state: %v", err) + return + } + if err := os.Rename(tmp, path); err != nil { log.Printf("saving the state: %v", err) } } diff --git a/examples/webhookbot/main.go b/examples/webhookbot/main.go new file mode 100644 index 0000000..3f2a82b --- /dev/null +++ b/examples/webhookbot/main.go @@ -0,0 +1,173 @@ +// Command webhookbot is a GoChat bot that takes its updates by webhook and +// answers over the HTTP Bot API: /ping answers pong. +// +// GOCHAT_URL=https://chat.example.com GOCHAT_BOT_TOKEN=b_... \ +// GOCHAT_WEBHOOK_SECRET=... go run ./examples/webhookbot -addr :8443 +// +// The server delivers only to https URLs on public addresses: put it behind +// a TLS-terminating proxy, or give it -cert and -key. setWebhook, with the +// same secret, points the bot at it; docs/bots.md shows how. +package main + +import ( + "bytes" + "context" + "crypto/subtle" + "encoding/json" + "errors" + "flag" + "fmt" + "io" + "log" + "net/http" + "os" + "os/signal" + "strings" + "syscall" + "time" + + "google.golang.org/protobuf/encoding/protojson" + "google.golang.org/protobuf/proto" + + pb "github.com/ssenerg/gochat/api/protobuf/gen/gochat/v1" + "github.com/ssenerg/gochat/pkg/gochat" +) + +// secretHeader carries the webhook's secret on each delivery. +const secretHeader = "X-GoChat-Bot-Secret" //nolint:gosec // a header name, not a secret + +var ( + // Updates are protobuf's JSON mapping; fields newer than this program + // are skipped. + jsonIn = protojson.UnmarshalOptions{DiscardUnknown: true} + jsonOut = protojson.MarshalOptions{UseProtoNames: true} +) + +func main() { + addr := flag.String("addr", ":8443", "the address to listen on") + cert := flag.String("cert", "", "a TLS certificate, to serve https itself") + key := flag.String("key", "", "the certificate's key") + flag.Parse() + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + b := &bot{ + api: strings.TrimRight(os.Getenv("GOCHAT_URL"), "/") + "/api/v2/bot/", + token: os.Getenv("GOCHAT_BOT_TOKEN"), + secret: os.Getenv("GOCHAT_WEBHOOK_SECRET"), + http: &http.Client{Timeout: 10 * time.Second}, + } + if err := run(ctx, b, *addr, *cert, *key); err != nil { + log.Fatal(err) + } +} + +func run(ctx context.Context, b *bot, addr, cert, key string) error { + if b.api == "/api/v2/bot/" || b.token == "" || b.secret == "" { + return errors.New("set GOCHAT_URL, GOCHAT_BOT_TOKEN and GOCHAT_WEBHOOK_SECRET") + } + me := &pb.MeResponse{} + if err := b.call(ctx, "getMe", &pb.MeRequest{}, me); err != nil { + return err + } + b.me = me.User + log.Printf("signed in as @%s, listening on %s", b.me.GetUsername(), addr) + + srv := &http.Server{Addr: addr, Handler: b, ReadHeaderTimeout: 10 * time.Second} + // Shutting down lets the delivery in progress finish. + down := make(chan struct{}) + stop := context.AfterFunc(ctx, func() { + defer close(down) + sctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second) + defer cancel() + _ = srv.Shutdown(sctx) + }) + var err error + if cert != "" { + err = srv.ListenAndServeTLS(cert, key) + } else { + err = srv.ListenAndServe() + } + if errors.Is(err, http.ErrServerClosed) { + <-down + return nil + } + stop() + return err +} + +type bot struct { + api, token, secret string + http *http.Client + me *pb.User +} + +// ServeHTTP takes one delivery. The next update waits for its answer, and +// one unanswered in 10 seconds comes again: answer quickly. +func (b *bot) ServeHTTP(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + w.WriteHeader(http.StatusMethodNotAllowed) + return + } + if subtle.ConstantTimeCompare([]byte(r.Header.Get(secretHeader)), []byte(b.secret)) != 1 { + w.WriteHeader(http.StatusUnauthorized) + return + } + body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 1<<20)) + if err != nil { + w.WriteHeader(http.StatusBadRequest) + return + } + // Any answer but 2xx has the server send the update again, holding + // back those after it: one the bot cannot handle is still taken. + ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second) + defer cancel() + ev := &pb.Event{} + if err := jsonIn.Unmarshal(body, ev); err != nil { + log.Printf("an update that does not parse: %v", err) + } else if err := b.handle(ctx, ev); err != nil { + log.Printf("update %d: %v", ev.GetPts(), err) + } + w.WriteHeader(http.StatusNoContent) +} + +func (b *bot) handle(ctx context.Context, ev *pb.Event) error { + m := ev.GetMessagePosted().GetMessage() + if m == nil || m.GetAuthor().GetId() == b.me.GetId() { + return nil + } + if name, _, ok := gochat.Command(m.Content, b.me.GetUsername()); !ok || name != "ping" { + return nil + } + return b.call(ctx, "sendMessage", &pb.PostMessageRequest{ChannelId: m.ChannelId, RootId: m.RootId, Content: "pong"}, &pb.PostMessageResponse{}) +} + +// call makes one call of the HTTP Bot API. +func (b *bot) call(ctx context.Context, method string, in, out proto.Message) error { + body, err := jsonOut.Marshal(in) + if err != nil { + return err + } + req, err := http.NewRequestWithContext(ctx, http.MethodPost, b.api+method, bytes.NewReader(body)) + if err != nil { + return err + } + req.Header.Set("Authorization", "Bearer "+b.token) + req.Header.Set("Content-Type", "application/json") + res, err := b.http.Do(req) + if err != nil { + return err + } + defer func() { _ = res.Body.Close() }() + var answer struct { + OK bool `json:"ok"` + Result json.RawMessage `json:"result"` + Description string `json:"description"` + } + if err := json.NewDecoder(io.LimitReader(res.Body, 4<<20)).Decode(&answer); err != nil { + return fmt.Errorf("%s: %s", method, res.Status) + } + if !answer.OK { + return fmt.Errorf("%s: %s (%d)", method, answer.Description, res.StatusCode) + } + return jsonIn.Unmarshal(answer.Result, out) +} diff --git a/examples/webhookbot/main_test.go b/examples/webhookbot/main_test.go new file mode 100644 index 0000000..e427221 --- /dev/null +++ b/examples/webhookbot/main_test.go @@ -0,0 +1,68 @@ +package main + +import ( + "io" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + + pb "github.com/ssenerg/gochat/api/protobuf/gen/gochat/v1" +) + +func TestDeliveries(t *testing.T) { + var ( + mu sync.Mutex + sent []*pb.PostMessageRequest + ) + api := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(r.Body) + in := &pb.PostMessageRequest{} + if r.URL.Path != "/api/v2/bot/sendMessage" || r.Header.Get("Authorization") != "Bearer b_test" || jsonIn.Unmarshal(body, in) != nil { + w.WriteHeader(http.StatusBadRequest) + _, _ = io.WriteString(w, `{"ok":false,"error_code":400,"description":"bad call"}`) + return + } + mu.Lock() + sent = append(sent, in) + mu.Unlock() + _, _ = io.WriteString(w, `{"ok":true,"result":{"message":{"id":"m2"}}}`) + })) + defer api.Close() + b := &bot{api: api.URL + "/api/v2/bot/", token: "b_test", secret: "s3cret", http: api.Client(), me: &pb.User{Id: "b1", Username: "ping_bot"}} + + post := func(content, author string) string { + return `{"pts":"7","pts_count":1,"message_posted":{"message":{"id":"m1","channel_id":"c1","author":{"id":"` + author + `"},"content":"` + content + `"}}}` + } + for _, tc := range []struct { + name, method, secret, body string + status, sent int + }{ + {"no secret", http.MethodPost, "", post("/ping", "u1"), http.StatusUnauthorized, 0}, + {"a wrong secret", http.MethodPost, "s3cres", post("/ping", "u1"), http.StatusUnauthorized, 0}, + {"a GET", http.MethodGet, "s3cret", "", http.StatusMethodNotAllowed, 0}, + {"a ping", http.MethodPost, "s3cret", post("/ping", "u1"), http.StatusNoContent, 1}, + {"a ping to the bot", http.MethodPost, "s3cret", post("/ping@ping_bot", "u1"), http.StatusNoContent, 2}, + {"a ping to another bot", http.MethodPost, "s3cret", post("/ping@other_bot", "u1"), http.StatusNoContent, 2}, + {"the bot's own", http.MethodPost, "s3cret", post("/ping", "b1"), http.StatusNoContent, 2}, + {"a field it does not know", http.MethodPost, "s3cret", `{"pts":"8","pts_count":1,"brand_new":{"x":1}}`, http.StatusNoContent, 2}, + {"no JSON", http.MethodPost, "s3cret", "nope", http.StatusNoContent, 2}, + } { + req := httptest.NewRequest(tc.method, "/", strings.NewReader(tc.body)) + if tc.secret != "" { + req.Header.Set(secretHeader, tc.secret) + } + w := httptest.NewRecorder() + b.ServeHTTP(w, req) + mu.Lock() + n := len(sent) + mu.Unlock() + if w.Code != tc.status || n != tc.sent { + t.Errorf("%s: answered %d and sent %d messages, want %d and %d", tc.name, w.Code, n, tc.status, tc.sent) + } + } + if m := sent[0]; m.ChannelId != "c1" || m.Content != "pong" { + t.Errorf("sent %v", m) + } +} diff --git a/test/e2e/bots_test.go b/test/e2e/bots_test.go index a829c0f..ac1dd75 100644 --- a/test/e2e/bots_test.go +++ b/test/e2e/bots_test.go @@ -1,11 +1,14 @@ package e2e import ( + "bytes" "context" "encoding/json" "errors" "net/http" "net/url" + "os" + "os/exec" "regexp" "strings" "testing" @@ -295,6 +298,36 @@ func testBots(t *testing.T, admin *client, adminWS *socket, teamID, town string, } }) + step("the Python example bot", func(t *testing.T) { + python, err := exec.LookPath("python3") + if err != nil { + t.Skip("no python3") + } + pctx, stop := context.WithCancel(ctx) + var out bytes.Buffer + cmd := exec.CommandContext(pctx, python, "-u", "../../examples/dicebot/dicebot.py") + cmd.Env = append(os.Environ(), "GOCHAT_URL="+baseURL, "GOCHAT_BOT_TOKEN="+token) + cmd.Stdout, cmd.Stderr = &out, &out + if err := cmd.Start(); err != nil { + t.Fatal(err) + } + defer func() { + stop() + _ = cmd.Wait() + if t.Failed() { + t.Logf("the bot's output:\n%s", out.String()) + } + }() + adminWS.send(map[string]string{"op": "post", "channel_id": town, "content": "/roll@" + name + " 20"}) + f := adminWS.expect(20*time.Second, "d20: ", `data-press="roll:20"`, `badge bot`) + m := msgIDRe.FindStringSubmatch(f) + if m == nil { + t.Fatalf("no message id in %s", f) + } + adminWS.send(map[string]string{"op": "press", "channel_id": town, "message_id": m[1], "data": "roll:20"}) + adminWS.expect(10*time.Second, "toast", "Rolled again.") + }) + step("resetting the token stops the bot", func(t *testing.T) { _, page := admin.hx("/settings/bots/"+botID+"/token", url.Values{}) m := botTokenRe.FindStringSubmatch(page)