Skip to content

fix(loop): detect a silent harness and hand the task to another one - #17

Merged
sreekar2403 merged 6 commits into
mainfrom
feat/oss-standout-p0p1
Sep 1, 2026
Merged

fix(loop): detect a silent harness and hand the task to another one#17
sreekar2403 merged 6 commits into
mainfrom
feat/oss-standout-p0p1

Conversation

@sreekar2403

@sreekar2403 sreekar2403 commented Sep 1, 2026

Copy link
Copy Markdown
Owner

What / Why

A harness that goes quiet used to cost a whole task. The observed run: the router sent work to pi with a local ollama model that never loaded. pi printed its session header, agent_start, turn_start — and then nothing. The run deadline is five minutes and only catches a CLI that is slow, so the task sat there, and when it finally gave up the loop's only move was to ask the same stuck binary again with a slightly reworded prompt. Ten iterations, no work done, no reason surfaced.

This adds a silence watchdog and a fallback to a different provider:

  • runHarness now watches for a gap with no byte on either stream. Every one of these CLIs prints something every few seconds while it is genuinely working, so a long gap is a reliable signal that nobody is coming. The run is stopped and reported as silent — a condition of its own, distinct from timedOut (ran slowly, deadline caught it) and from a plain non-zero exit (ran, failed, said why).
  • LoopEngine treats silence as "this harness never attempted the task" rather than a failed attempt: it drops the harness for the rest of the run, drops the harness and model pins, and routes again over what is left. The prompt is passed through untouched — the next CLI gets the original request, not one with a note about somebody else's failure stapled to it. When every harness has gone silent it stops and says so instead of burning the remaining iterations.

It also fixes a related silent failure. Several CLIs report a failure as an error event on stdout and exit non-zero with stderr empty — opencode answering a dead model with "Unexpected server error" is the case that prompted it. That message reached the activity trail and nowhere else: the loop's retry check reads stderr and the chat window reads output, and both were blank. The run failed, said exactly why, and nobody was listening. The error text is now lifted out of the event stream into both.

Two knobs in Settings → Execution: loop.idleTimeoutMs (default 120s, 0 disables) and loop.harnessFallback (default on).

Review fixes in 536308f:

  • A clean exit is never silent. silent was computed from output alone and checked before success, so a harness that exited 0 while saying almost nothing had its succeeded run discarded and the task handed to a second CLI to redo.
  • The exit fallback now arms only for runs we killed. It exists for the Windows case where a killed shim leaves a grandchild holding the pipe and close never fires (the run would hang forever holding a ConcurrencyGate slot); armed for every run it raced a healthy child still draining its stream.
  • loop.idleTimeoutMs and loop.harnessFallback are validated on PUT /api/settings like every other loop key.

How tested

  • pnpm lint
  • pnpm exec tsc --noEmit
  • pnpm exec vitest run — 406 server tests pass
  • hive doctor

New coverage, against real subprocesses rather than mocks — the behaviour under test is what happens when pipes go quiet, which a mock cannot reproduce:

  • runner.test.ts — a child that prints a header then nothing is stopped and reported silent; a slow-but-talking child is left alone; a quiet successful run is not called silent; a cancelled run is not called silent; idleTimeout: 0 disables the check; an error-only run reaches stderr and output.
  • loopEngine.test.ts — the task moves to another harness rather than back to the silent one; the dead harness's model pin does not travel with it; the loop stops once every harness is silent (2 attempts, not maxIterations); the prompt is unchanged across the switch; harnessFallback: false is respected.

Breaking changes

None. Both config fields are new with defaults, and deepMerge over createDefaultConfig() backfills them for configs written before they existed.

Closes

N/A — found in a real run rather than filed.

sreekar2403 and others added 5 commits August 28, 2026 19:44
…e hygiene

- .gitignore: keep docs/ tracked (ignore only docs/superpowers), re-ignore per-machine hive.config.json + docs/ToDo.md
- install.sh: fix repo URL sreekar2403/hive, use pnpm, update next steps (CLIs bring own auth)
- Add OSS community files: CONTRIBUTING, CoC, SECURITY, issue/PR templates
- CI: split lint-typecheck (lint + pnpm typecheck + prettier --check), test matrix ubuntu/windows x node 20/22, coverage via v8, threshold 60
- Build: pnpm build now builds server+client, add build:server/client, format:check, typecheck (per-package), engines >=22, add helmet/rate-limit/zod deps
- Server hardening: helmet, rate-limit (/api 120/min, /api/chat 20/min), zod validation for /api/chat (20k max), pagination on GET /api/tasks (limit/offset, max 100)
- Lint: fix branches.test escape, server.ts unused imports, store.ts LAYOUT
- Docs: README fan-out/attachments/vision sections + ports & API base + repository layout, DEVELOPMENT build + fan-out/attachments/vision + ports, new docs/ARCHITECTURE.md
- Vitest: projects server(node) + client(jsdom), coverage v8, add jsdom + coverage deps
- Release: CHANGELOG, dependabot, Dockerfile, devcontainer, release workflow
- Format: prettier --write across repo (now CI-clean)

Co-authored-by: opencode
…lidate new config

Three review fixes on the silence-detection work.

A clean exit is never silent. `silent` was computed from output alone, and
the loop checked it before checking success, so a harness that exited 0
while saying almost nothing had its succeeded run discarded and the task
handed to a second CLI to redo. Exit code now decides it, and the loop
declines to reroute a successful run either way.

The 'exit' fallback only arms for runs we killed. It exists for the
Windows case where a killed shim leaves a grandchild holding the pipe and
'close' never fires; armed for every run it raced a healthy child still
draining, and anything arriving after the 2s timer was dropped. A child
exiting on its own now always gets to finish draining.

loop.idleTimeoutMs and loop.harnessFallback are validated like every
other loop key, so a bad PUT is rejected instead of persisted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sreekar2403 sreekar2403 changed the title Feat/oss standout p0p1 fix(loop): detect a silent harness and hand the task to another one Sep 1, 2026
…ve build

Three CI failures, all of them failing on main as well and none of them
about the code under review.

Typecheck never built @hive/shared. The server's tsconfig `references` it,
so on a clean checkout every import of it was TS6305 — "output file has not
been built from source" — and the cascade made ordinary shared types look
like errors in router.ts and attachments.ts. `--noEmit` on a composite
project cannot produce the dist the reference needs; `--build` can.

Node 20 is not a platform this project supports: package.json already
declares `engines.node: >=22`, and the matrix contradicted it. The test
environment agreed — jsdom 30 pulls undici 8, which calls
`webidl.util.markAsUncloneable`, absent on Node 20, so all seven client
test files died at worker startup and the job failed on coverage rather
than on anything real. Matrix is 22 and 24 now.

The windows runners ship Visual Studio 18, which the node-gyp pnpm vendors
does not recognise: it logs `unknown version "undefined"`, concludes no
Visual Studio is installed, and better-sqlite3's source build takes
`pnpm install` down before a single test runs. Point npm_config_node_gyp
at a current node-gyp on Windows only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sreekar2403
sreekar2403 merged commit 6eb212b into main Sep 1, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant