GitHub pull_request.labeled event did not trigger automation run

[all-hands-bot]

Summary

A GitHub event-triggered automation configured for OpenHands/software-agent-sdk did not create any automation runs when a test PR was labeled with oh-cloud-review.

The GitHub label events were successfully emitted by GitHub, and local inspection of the automation service code indicates the automation trigger/filter should match a representative pull_request.labeled payload. However, the automation service shows zero runs, which suggests the event is not reaching the automation service or is being routed to a different OpenHands org before trigger matching.

Test setup

Automation created via the plugin preset:

• Automation name: PR Review: software-agent-sdk oh-cloud-review
• Automation ID: 5938f464-b585-429f-a95a-8f0e12a3a02c
• Automation org ID: 8e0a0d06-afbe-41b0-ac49-ebfbc774580e
• Source: github
• Plugin: github:OpenHands/extensions, repo_path: plugins/pr-review, ref: main
• Target repo: OpenHands/software-agent-sdk
• Test PR: Test oh-cloud-review automation trigger software-agent-sdk#3205
• Label: oh-cloud-review

Original trigger:

{
  "type": "event",
  "source": "github",
  "on": "pull_request.labeled",
  "filter": "repository.full_name == 'OpenHands/software-agent-sdk' && label.name == 'oh-cloud-review'"
}

After debugging, the trigger was broadened to include issues.labeled as well, in case PR label events were normalized that way:

{
  "type": "event",
  "source": "github",
  "on": ["pull_request.labeled", "issues.labeled"],
  "filter": "repository.full_name == 'OpenHands/software-agent-sdk' && label.name == 'oh-cloud-review' && (pull_request.number != null || issue.pull_request.url != null)"
}

What happened

1. Created a temporary PR in OpenHands/software-agent-sdk:
   • Test oh-cloud-review automation trigger software-agent-sdk#3205
2. Added the label oh-cloud-review.
3. Re-applied the label multiple times to emit fresh label events.
4. Polled the automation runs endpoint repeatedly:

GET /api/automation/v1/5938f464-b585-429f-a95a-8f0e12a3a02c/runs?limit=10

Response remained:

{
  "runs": [],
  "total": 0
}

The automation detail also still showed no visible run activity.

Evidence that GitHub emitted the label event

GitHub issue events for PR #3205 showed label activity, including:

labeled    2026-05-11T15:24:55Z    oh-cloud-review    all-hands-bot
unlabeled  2026-05-11T15:30:18Z    oh-cloud-review    all-hands-bot
labeled    2026-05-11T15:30:22Z    oh-cloud-review    all-hands-bot

GitHub repo events also showed PullRequestEvent entries for PR #3205 with action=labeled and payload keys similar to:

['action', 'number', 'pull_request', 'label', 'labels']

So this does not look like a GitHub-side label operation failure.

Evidence that trigger matching should work

I cloned OpenHands/automation and inspected:

• openhands/automation/event_router.py
• openhands/automation/event_schemas/github.py
• openhands/automation/trigger_matcher.py
• openhands/automation/filter_eval.py
• openhands/automation/utils/webhook.py

A representative payload for the observed event parses as:

event_key = pull_request.labeled

and matches both the original trigger and the broadened trigger locally:

from openhands.automation.event_schemas import parse_event
from openhands.automation.trigger_matcher import matches_trigger
from openhands.automation.schemas import EventTrigger

payload = {
    "action": "labeled",
    "number": 3205,
    "label": {"name": "oh-cloud-review"},
    "pull_request": {
        "number": 3205,
        "title": "Test PR",
        "state": "open",
        "draft": False,
        "merged": False,
        "base": {"ref": "main", "sha": "a"},
        "head": {"ref": "test", "sha": "b"},
        "labels": [{"name": "oh-cloud-review"}],
        "user": {"id": 1, "login": "u", "type": "User"},
    },
    "repository": {
        "id": 1,
        "name": "software-agent-sdk",
        "full_name": "OpenHands/software-agent-sdk",
        "private": False,
        "owner": {"login": "OpenHands", "type": "Organization", "id": 2},
    },
    "sender": {"id": 182006032, "login": "all-hands-bot", "type": "User"},
}

event = parse_event("github", payload)
assert event.event_key == "pull_request.labeled"

trigger = EventTrigger.model_validate({
    "type": "event",
    "source": "github",
    "on": "pull_request.labeled",
    "filter": "repository.full_name == 'OpenHands/software-agent-sdk' && label.name == 'oh-cloud-review'",
})

assert matches_trigger(trigger, "github", event.event_key, payload) is True

Therefore the most likely failure is upstream of matches_trigger().

Relevant implementation path

From OpenHands/OpenHands, GitHub webhooks are received at:

enterprise/server/routes/integration/github.py

The route only forwards to the automation service when:

if AUTOMATION_EVENT_FORWARDING_ENABLED:
    background_tasks.add_task(
        automation_event_service.forward_event,
        provider=ProviderType.GITHUB,
        payload=payload_data,
        installation_id=installation_id,
    )

The forwarding service is:

enterprise/server/services/automation_event_service.py

The forwarding path:

1. Extracts owner info from payload.repository.owner.login.
2. Resolves the GitHub owner to an OpenHands org via OrgGitClaim.
3. For org repos, if the GitHub org is not claimed, it skips forwarding.
4. If resolved, it POSTs to:

{AUTOMATION_SERVICE_URL}/v1/events/{org_id}/{provider}

with an internal HMAC signature using AUTOMATION_WEBHOOK_SECRET.

In OpenHands/automation, the event is then received at:

/api/automation/v1/events/{org_id}/github

Implemented in:

openhands/automation/event_router.py

Suspected causes

1. Event forwarding may be disabled or misconfigured in the OpenHands server

The server-side route only forwards if:

AUTOMATION_EVENT_FORWARDING_ENABLED=true
AUTOMATION_SERVICE_URL is configured
AUTOMATION_WEBHOOK_SECRET is configured

The chart defaults appear to have forwarding disabled:

automationService:
  url: ""
  eventForwardingEnabled: false

If production has not overridden these, GitHub webhooks can still be processed by the normal GitHub integration, but automations will never see them.

2. github/openhands may not be claimed by the OpenHands org that owns the automation

The automation belongs to OpenHands org:

8e0a0d06-afbe-41b0-ac49-ebfbc774580e

For org-owned repos, forwarding requires an OrgGitClaim mapping roughly equivalent to:

provider = github
git_organization = openhands
org_id = 8e0a0d06-afbe-41b0-ac49-ebfbc774580e

If the claim is missing, the forwarding code skips the event.

If the claim exists but points to another OpenHands org, the event will be forwarded to that org and the automation above will not see it.

I tried to query:

GET /api/organizations/8e0a0d06-afbe-41b0-ac49-ebfbc774580e/git-claims

but my token received:

403 Requires manage_org_claims permission

so I could not verify the claim.

3. Stale negative Redis cache could cause forwarding to continue skipping events

The forwarding code caches negative org-claim resolution results for one hour:

automation:org_claim:github:openhands = none
TTL = 3600 seconds

If the first test happened before github/openhands was claimed, subsequent tests may continue to skip forwarding until the cache expires or is cleared.

4. GitHub App delivery/subscription issue

The GitHub App should be checked for a pull_request webhook delivery for PR #3205, action labeled, and whether /integration/github/events returned 200.

The GitHub App manifest in OpenHands-Cloud appears to subscribe to:

[
  "issue_comment",
  "pull_request",
  "pull_request_review_comment"
]

so pull_request.labeled should be included, but the actual installed app delivery logs are the source of truth.

5. Background task may not run if normal GitHub handling raises before response

The GitHub route adds forwarding as a FastAPI background task, then awaits normal resolver handling:

background_tasks.add_task(...)
await github_manager.receive_message(message)

If github_manager.receive_message() raises, the route returns 400 and the background task may not execute. This seems less likely for a plain pull_request.labeled event, but logs should confirm.

Small implementation observation

last_triggered_at may not be updated for event-triggered automations.

Manual/cron dispatch uses create_pending_run(), which updates Automation.last_triggered_at.

Event dispatch uses create_automation_run() in:

openhands/automation/utils/webhook.py

That creates an AutomationRun, but does not update Automation.last_triggered_at.

So last_triggered_at: null alone is not proof of no event match. In this case, however, runs: [] confirms no run was created.

Suggested debugging steps

Could someone with production access check the following around the label timestamps for PR #3205?

1. OpenHands server env:

AUTOMATION_EVENT_FORWARDING_ENABLED
AUTOMATION_SERVICE_URL
AUTOMATION_WEBHOOK_SECRET

2. Org claim:

github / openhands -> 8e0a0d06-afbe-41b0-ac49-ebfbc774580e

3. Redis cache key:

automation:org_claim:github:openhands

4. GitHub App delivery logs for:

repo: OpenHands/software-agent-sdk
PR: #3205
event: pull_request
action: labeled
label: oh-cloud-review

5. OpenHands server logs for:

[AutomationEventService] Forwarded github event to org ...
[AutomationEventService] github org OpenHands not claimed ...
[AutomationEventService] Automation service returned ...
Error processing GitHub event

6. Automation service logs for:

Received github event: key=pull_request.labeled
Event matched 1/... automations

Expected behavior

Adding oh-cloud-review to a PR in OpenHands/software-agent-sdk should create a pending automation run for the configured automation.

Actual behavior

No run was created:

{
  "runs": [],
  "total": 0
}

Impact

GitHub-event-triggered automations may silently fail for org-owned repositories if forwarding is not enabled/configured, org claims are missing/misrouted, or negative claim results are cached.

---

This issue was created by an AI agent (OpenHands) on behalf of the user.

[xingyaoww]

Datadog Investigation Results

This comment was created by an AI agent (OpenHands) on behalf of the user.

Root Cause Confirmed: Missing OrgGitClaim for GitHub org OpenHands (Suspected Cause #2)

The Datadog logs definitively confirm that the github/OpenHands org is not claimed in the OpenHands Cloud database, which causes the AutomationEventService to silently drop all webhook events from OpenHands-owned repositories before they reach the automation service.

---

Evidence from Datadog Logs

1. Events were dropped at the forwarding layer (service:deploy)

During the label event window (~15:24-15:30 UTC), 250+ log entries show:

[AutomationEventService] github org OpenHands not claimed and no personal org found, skipping

These appear continuously from at least 15:00 through 16:52+ UTC, and still persisted at time of investigation (17:27 UTC). This confirms GitHub events from OpenHands/software-agent-sdk never reached the automation service.

2. No pull_request.labeled events reached the correct org

Searching for pull_request.labeled events for org 8e0a0d06-afbe-41b0-ac49-ebfbc774580e returned 0 results across the entire investigation window. The automation service never saw the label events.

3. Event forwarding IS working for other orgs

Many other pull_request.labeled events were successfully received by the automation service during the same time window — but routed to other orgs (e.g., 47d83138, 7a3a170e, f5164aca, etc.). The forwarding infrastructure itself is working; only the OpenHands org mapping is missing.

4. Partial claim fix observed around ~16:51 UTC

Starting at 16:51:57 UTC, some events began being forwarded to org 8e0a0d06:

• Two push events got 200 responses with "Event matched 0/1 automations" (expected — the automation listens for pull_request.labeled, not push)
• Multiple workflow_job/workflow_run events got 400 responses (unsupported event types in the automation service)

However, "not claimed" messages continue simultaneously on other server pods, indicating a Redis negative cache issue (Suspected Cause #3). Some server instances still have a cached none result for automation:org_claim:github:openhands (TTL 3600s).

5. No new pull_request.labeled events occurred after the partial fix

The original label events at 15:24-15:30 UTC were permanently lost. After the claim was partially fixed (~16:51), no new pull_request.labeled events were emitted, so the automation still has 0 runs.

---

Timeline Summary

Time (UTC)	Event
15:00+	"github org OpenHands not claimed" messages continuously logged
15:24:55	Label oh-cloud-review added to PR #3205 — dropped by forwarding service
15:30:22	Label re-applied — dropped again
~16:51	Claim appears to have been partially created for github/OpenHands → 8e0a0d06
16:52	Some events start flowing to correct org (push/CI events only, no label events)
16:52+	"Not claimed" messages still appear on some server pods (stale negative Redis cache)
17:27	"Not claimed" messages still appearing — cache hasn't fully expired

---

Confirmed Causes

#	Cause	Status
#2	OrgGitClaim missing for github/OpenHands → 8e0a0d06	✅ Confirmed as primary root cause
#3	Stale negative Redis cache continuing to drop events	✅ Confirmed as secondary cause (events still being dropped at 17:27)
#1	Event forwarding disabled	❌ Not the issue — forwarding works for other orgs
#4	GitHub App delivery issue	❌ Not the issue — events were received by the server
#5	Background task not running	❌ Not the issue — forwarding code ran but skipped

---

Recommended Actions

1. Verify the OrgGitClaim exists and is correctly mapping github/OpenHands → org 8e0a0d06-afbe-41b0-ac49-ebfbc774580e
2. Flush the Redis negative cache key automation:org_claim:github:openhands across all server pods to immediately unblock event forwarding
3. Re-trigger the label event on PR #3205 (remove and re-add oh-cloud-review) to test end-to-end after the claim and cache are fixed
4. Consider reducing the negative cache TTL (currently 3600s / 1 hour) for org claim lookups to minimize the window of silent event loss after a claim is created

[all-hands-bot]

Follow-up after retrying now that OpenHands is claimed

The original routing problem is fixed: after the GitHub org claim was added, re-applying oh-cloud-review to OpenHands/software-agent-sdk#3205 did create automation runs.

Observed runs:

• 378037af-ce89-457c-b79b-d4856c3e4d1b — event was received, run started, then failed.
• 3695b246-b57a-4c90-9dd5-44498b915ae3 — event was received, run started, then failed.

Both failed with:

Server error '500 Internal Server Error' for url 'http://localhost:60000/api/conversations'

Additional investigation

The automation record currently uses:

{
  "tarball_path": "oh-internal://uploads/6e382912-f763-4491-9222-cbc3f71fe856",
  "setup_script_path": "setup.sh",
  "entrypoint": "python main.py"
}

I inspected the extracted tarball from a later diagnostic run. It is generated by an older plugin preset:

SDK_VERSION="1.19.1"
pip install -q --no-cache-dir   "openhands-sdk==${SDK_VERSION}"   "openhands-tools==${SDK_VERSION}"   "openhands-workspace==${SDK_VERSION}"

and the entrypoint is python main.py.

This differs from the current code on main in this repo, which appears to have already moved to an isolated venv and a venv entrypoint:

VENV_ENTRYPOINT = ".venv/bin/python main.py"

and setup scripts using uv venv .venv with SDK 1.21.1.

I also created and immediately deleted a temporary plugin-preset automation to inspect what production currently generates. It still came back with:

{
  "entrypoint": "python main.py",
  "setup_script_path": "setup.sh"
}

So production appears to still be generating the older preset, not the current main preset implementation.

Important note about the existing automation

The automation tarball is immutable (oh-internal://uploads/...), so existing automations do not automatically pick up preset fixes from this repo. Even after the preset code is fixed/deployed, this automation would need to be recreated or otherwise have its tarball regenerated.

I briefly tried changing only the existing automation entrypoint to .venv/bin/python main.py, but that is not compatible with the old tarball because its setup script does not create .venv; the sandbox command failed with:

/bin/sh: 1: .venv/bin/python: not found

I restored the automation entrypoint back to python main.py afterward.

Current hypothesis

There are now two separate issues:

1. Original issue fixed: missing OrgGitClaim prevented GitHub events from reaching the automation service.
2. Current runtime issue: production plugin-preset automations are still generated with the older SDK/setup/entrypoint. That old generated script fails during SDK conversation creation against the sandbox agent server (POST /api/conversations).

Suggested next steps

1. Deploy the current preset implementation from this repo, or otherwise update production so new plugin-preset automations generate the venv-based setup/entrypoint.
2. Recreate the affected automation after the deploy so it gets a new tarball.
3. Consider adding a preset/version field to generated automations or tarballs so stale generated automation code is visible in the API response.
4. Consider surfacing sandbox bash output in automation run errors when the SDK completion callback is not sent. In the .venv/bin/python diagnostic run, the bash command had already exited with 127, but the automation run remained RUNNING until watchdog/timeout because no callback occurred.

This comment was created by an AI agent (OpenHands) on behalf of the user.