-
Notifications
You must be signed in to change notification settings - Fork 37
fix(openhands-automation): warn about webhook limitations on local deployments, recommend polling #274
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(openhands-automation): warn about webhook limitations on local deployments, recommend polling #274
Changes from all commits
fec8ca3
c74ba92
ba4e984
0be327b
548f41b
f5b6c37
6df73f0
e924ee4
94385d2
6f32842
84881a2
cbe6ee7
d3d2170
e8537e0
fbeec8b
a10e9f7
e14f740
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -27,11 +27,19 @@ Two components work together to run automations: | |
| **Automation Service** (API at `OPENHANDS_HOST/api/automation/v1`) | ||
| Manages the *when*: holds automation definitions, schedules cron-triggered runs, dispatches webhook-triggered runs, and receives completion callbacks to mark runs as done. This is the API you call to create, update, and manage automations. | ||
|
|
||
| **Agent Server** (reachable at `AGENT_SERVER_URL` inside a run) | ||
| Manages the *what*: the runtime environment where automation scripts execute and where conversations (AI agent interactions with tools, bash, file editing, etc.) run. When a run is triggered, the automation service uploads the automation's tarball to the agent server, which unpacks and runs the entrypoint script. The script runs inside the agent server and connects back to it using `AGENT_SERVER_URL` and a session API key to start, monitor, and stop conversations. | ||
| **Agent Server** (accessible as `AGENT_SERVER_URL` inside script runs) | ||
| Manages the *what*: the runtime environment where automation scripts execute and where conversations (AI agent interactions with tools, bash, file editing, etc.) run. When a run is triggered, the automation service uploads the automation's tarball to the agent server, which unpacks and runs the entrypoint script. The script connects back to the agent server using `AGENT_SERVER_URL` and a session API key to start, monitor, and stop conversations. | ||
|
|
||
| The agent server typically runs inside a **sandbox** (a Docker or Kubernetes container). Some deployments use sandboxless mode, where the agent server runs directly on a host. | ||
|
|
||
| **Key environment variables:** | ||
|
|
||
| | Variable | Availability | Description | | ||
| |---|---|---| | ||
| | `RUNTIME_URL` | Ambient in cloud environments | Public-facing URL of the **agent server** sandbox. Use this to determine whether external webhook delivery is possible — if unset or local, webhooks cannot be received. The automation service may run at a separate URL (see Determining the API Host). | | ||
| | `AGENT_SERVER_URL` | Injected into scripts at run time only | Internal URL of the agent server. Available inside script execution context; **not** an ambient environment variable outside of a running script. | | ||
|
tofarr marked this conversation as resolved.
|
||
| | `OPENHANDS_HOST` | Shell convention only — set manually | Base URL for the automation service API. **Not a real environment variable.** Set it from the `<HOST>` system-prompt value, or default to `https://app.all-hands.dev`. Used in all `curl` examples throughout this skill. | | ||
|
|
||
| > **⚠️ CRITICAL — Agent behavior rules:** | ||
| > | ||
| > 0. **Does this task need an LLM at all? Check first.** Before picking a preset, ask whether the task actually requires reasoning, judgment, summarization, or open-ended tool use. If it is fully deterministic — fixed data transforms, scheduled HTTP calls, healthcheck pings, file rotation, picking from a known list, posting a templated message — an LLM-driven preset is overkill. Every run will consume LLM tokens, which adds up fast at high frequencies (every 5 min ≈ 288 runs/day). Surface the trade-off to the user and offer the custom-script path (see `references/custom-automation.md`) as the cheaper, more reliable option. Be especially careful for cron schedules tighter than hourly. | ||
|
|
@@ -53,6 +61,9 @@ The agent server typically runs inside a **sandbox** (a Docker or Kubernetes con | |
| > - **Custom script** — full control over code, with or without LLM; point them to `references/custom-automation.md` | ||
| > - Let the user choose which approach to use. | ||
| > 4. **Only create custom scripts after the user agrees to that path.** Refer to `references/custom-automation.md` for the full reference. | ||
| > 5. **Before suggesting event-triggered (webhook) automations, check whether the deployment is publicly reachable.** Check `RUNTIME_URL`. Webhooks require an internet-accessible URL so that external services (GitHub, Slack, Linear, etc.) can deliver events to the automation service. If `RUNTIME_URL` is unset, empty, or resolves to a local or private address (`localhost`, `127.0.0.1`, `0.0.0.0`, or any RFC 1918 range: `10.x.x.x`, `192.168.x.x`, `172.16–31.x.x`), the service cannot receive inbound webhook traffic from the public internet. In that case: | ||
| > - **Recommend a cron-based polling automation instead.** Have the automation run on a schedule and call the external service's API (e.g., the GitHub REST API) to check for new events since the last run. | ||
| > - Explain the limitation clearly to the user: "Because this is a local deployment, external services can't reach the webhook endpoint. I'll set up a polling automation using a cron schedule instead." | ||
|
|
||
| ### No-LLM Script Helpers | ||
|
|
||
|
|
@@ -102,7 +113,14 @@ All requests require Bearer authentication: | |
|
|
||
| **Before making API calls, determine the correct host:** | ||
|
|
||
| Look for a `<HOST>` value in the system prompt. If present, use that URL. Otherwise, default to `https://app.all-hands.dev`. | ||
| The automation service may run at a different URL from the agent server. In the examples throughout this skill, `${OPENHANDS_HOST}` is a shell-variable convention for the automation service base URL — it is **not** a real environment variable. Set it from context before running any curl command: | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why isn’t this a real env var? Maybe it can be?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is not a real env var in local because it is not needed locally. In the cloud / staging environment, it is not yet defined - but we may change that.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think maybe I should clarify: this reads a bit confusing. Why have a var which is set to the value of |
||
|
|
||
| - Look for a `<HOST>` value in the system prompt. If present, use that URL. | ||
| - Otherwise default to `https://app.all-hands.dev`. | ||
|
|
||
| ```bash | ||
| OPENHANDS_HOST="https://app.all-hands.dev" # replace with <HOST> if provided | ||
| ``` | ||
|
|
||
|
|
||
| ### Automation Endpoints | ||
|
|
@@ -138,7 +156,7 @@ Automations support two trigger types: | |
| | Trigger Type | Use Case | | ||
| |--------------|----------| | ||
| | **Cron** | Run on a schedule (daily, weekly, hourly, etc.) | | ||
| | **Event** | Run when a webhook event occurs (GitHub PR opened, issue commented, etc.) | | ||
| | **Event** | Run when a webhook event occurs (GitHub PR opened, issue commented, etc.) — **requires a publicly reachable deployment** | | ||
|
|
||
| --- | ||
|
|
||
|
|
@@ -266,6 +284,21 @@ curl -X POST "${OPENHANDS_HOST}/api/automation/v1/preset/prompt" \ | |
|
|
||
| --- | ||
|
|
||
| ## Polling as a Webhook Alternative | ||
|
tofarr marked this conversation as resolved.
|
||
|
|
||
| When the deployment cannot receive inbound webhook traffic (see rule 5), use a cron-triggered automation that calls the external service’s API on a schedule to check for new events. | ||
|
|
||
| ### Polling vs. Webhooks at a Glance | ||
|
|
||
| | | Webhooks (Event trigger) | Polling (Cron trigger) | | ||
| |---|---|---| | ||
| | **Requires public URL** | Yes | No — works locally | | ||
| | **Latency** | Near-instant | Up to one poll interval | | ||
| | **API calls** | Only on real events | Every poll interval | | ||
| | **Best for** | Cloud / public deployments | Local or private deployments | | ||
|
|
||
|
tofarr marked this conversation as resolved.
tofarr marked this conversation as resolved.
|
||
| --- | ||
|
|
||
| ## Event-Triggered Automations (Webhooks) | ||
|
tofarr marked this conversation as resolved.
tofarr marked this conversation as resolved.
|
||
|
|
||
| Event-triggered automations run when a webhook event occurs — like a GitHub PR being opened, an issue receiving a comment, or a custom service sending a notification. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this accurate? Is it cloud-only and if so, could we check that it’s clear that it’s cloud-only?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is accurate as of earlier this morning. I think in the long run we should unify these variables across projects, but that's a separate PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I guess I should clarify: the text says
The text for the first var talks about local.
The text for the second var doesn't specify anything, and it's in reality cloud-only. (?)
This reads very confusing. People will read that the second var as being for local too.