5/6 backends fail trivial task on fresh self-host (1/6 serve)

[yakimoto]

Fresh Community Edition, Docker, image harnessrouter/harnessrouter:latest (pulled 2026-09-17). Google provider wired headlessly via connection+policies (gemini 8/8 + opencode 8/53 + hermes/qwen/kimi/pi 8 available in catalog).

Same trivial task + same model through 6 backends (Reply with exactly: HR-<BACKEND>-OK, gemini-3.7-flash):

backend	result
gemini	done, 3.8s, exact output
opencode	failed x2, 322.7s, empty result
hermes	failed, 322.7s, empty
qwen	failed, 322.7s, empty
kimi	failed, empty
pi	failed, 2.5s, empty

Two notes:

1. The identical 322.7s across four backends smells like a systemic timeout, not task variance.
2. Direct opencode run inside the container fails fast with UnknownError ... ref err_f77d16ec, so at least the opencode failure is CLI-level (gateway reports it faithfully as failed+empty — good). The 322s hangs never surface a cause.

Ask: is there a supported way to see WHY a backend run failed (which log, which trace field)? The traces list shows status failed with empty error/result, which makes self-host debugging a wall. Happy to retest anything.

[yakimoto]

Follow-up with funded keys wired headlessly (env connections+policies): OpenAI key -> codex/opencode policies, Mistral-compatible endpoint -> cline policy, Google key -> gemini (was already working).

Wider matrix, same trivial echo task:

backend	result
gemini	done, 3.8s, exact
codex (openai key)	failed, 5.5s, empty
cline (mistral endpoint)	failed, 2.5s, empty
opencode (openai key)	timed out, empty
opencode (google key)	failed, 322.7s, empty (x2)
hermes/qwen/kimi/pi (google key)	failed

1/9 serve. The traces list shows status failed with empty error/result, so there is still nothing to debug with. The fastest fix on your side would be surfacing the backend CLI's stderr in the trace error field — right now self-hosters cannot distinguish bad credentials from a broken backend without docker exec.

[yakimoto]

One more datum for the observability ask: the gateway container logs contain zero error/fail/exception lines across all of these failed runs. The failure is silent in the traces API AND in server logs — there is currently no surface anywhere that says why a backend run died. Even a backend-exit-code + last-stderr-line in the trace would unblock self-host debugging.

[yakimoto]

Decisive datapoint from the Console UI (same task that fails silently via API): the console shows The free trial covers agent work and memory; model usage runs on your own key. Add a provider key under Bring Your Own Key to run this task. So the cause EXISTS in the system — the Console surfaces it, while the API traces list and server logs show nothing. The gap is precisely: backend failure reasons reach the UI but not the API/trace surface. Surfacing that string (or the backend exit detail) in the trace error field would close it.

[yakimoto]

Preempting the billing question: re-ran codex + opencode against a funded key (xAI grok-4.6, credit verified with a direct API call first). Both fail fast (0.1-1.5s) with empty errors. Also confirmed OpenAI key is insufficient_quota-dead and Mistral is rate-limited, so earlier OpenAI-path failures carry that asterisk — but the xAI runs isolate backend behavior from billing, and the verdict holds: 1/9 serve.