hermes: the first Python harness — and the streaming exchange orca was dropping - #33
hermes: the first Python harness — and the streaming exchange orca was dropping#33xizhuomengcontin wants to merge 4 commits into
Conversation
8399fb1 to
5483933
Compare
![orcacode-review[bot]](https://avatars.githubusercontent.com/in/4254738?s=60&v=4){kind=small}
There was a problem hiding this comment.
⚠️ Outdated review — the PR head changed from 8399fb1 to 5483933 during the review; these findings describe a diff the PR no longer has.
These findings describe8399fb18773e. Re-run the review to check the PR as it stands now.
🐳 OrcaCode Review
✅ No findings — nothing to flag in this PR. Great work!
OrcaCode Review — Route Smarter. Ship Safer. Spend Less.
Engine-reported: 280 calls · 14.2M tokens · 98% cached
❤️ Share · Install OrcaCode Review
Free on GitHub — the review runs on your own OrcaRouter key. If it helped, a shout-out goes a long way.
Share: X · Reddit · LinkedIn
Follow: X · Discord · LinkedIn · OrcaRouter
There was a problem hiding this comment.
🐳 OrcaCode Review
✅ No findings — nothing to flag in this PR. Great work!
OrcaCode Review — Route Smarter. Ship Safer. Spend Less.
Engine-reported: 304 calls · 16.6M tokens · 98% cached
❤️ Share · Install OrcaCode Review
Free on GitHub — the review runs on your own OrcaRouter key. If it helped, a shout-out goes a long way.
Share: X · Reddit · LinkedIn
Follow: X · Discord · LinkedIn · OrcaRouter
An intercepted streaming response was recorded only when it reached `end`. A harness that stops reading the moment it has its answer never gets there, so the run kept nothing at all -- and then reported `capture.empty` with "the agent never called the proxy", about a request orca had itself decrypted and forwarded. That message is the opposite of what happened, which is the part that made this expensive to find. Hermes is the harness that showed it. Its answer arrives over SSE and its client leaves as soon as the answer is complete; the proxy had the request body, the response headers and a prefix of the body in hand, and dropped all three. Instrumenting the CONNECT handler is what settled it: three connections, all `allowed=true`, two `POST /zen/v1/chat/completions` arriving at the intercepting server with no TLS error, and zero events in the trace. So both forwarders now hoist what they have read out of the response handler and record it once, from whichever path settles first. `NetExchange` gains `abandoned`, which is a different fact from `responseTruncated`: one says the agent stopped asking for bytes, the other says orca stopped keeping them. The h2 path had the same gap -- its `abandon()` destroyed the upstream and resolved without recording -- and is fixed the same way. One follow-on change, in `server.ts`: a response-less call is no longer promoted to a model exchange. A POST abandoned before the origin answered satisfies the dialect check on its path alone, and promoting it puts an entry in the replay set that can answer nothing and cannot be forked. It is kept as network traffic, which is what it is. In a real Hermes run this is the difference between one clean model exchange plus one recorded probe, and three model exchanges of which two are empty. The client-gone signal is `res.destroyed` and deliberately not `req.destroyed`. A request with no body destroys itself once read, so asking about `req` turned every upstream failure -- an origin certificate orca refuses to trust, most of all -- into a silent abandonment with no 502 for the client to see. The repo's own test for that refusal is what caught it. The three new tests were mutation-checked: reverting the client-gone listener, the h2 `emit`, or the `status !== 0` guard each turns exactly the matching test red. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every other adapter here fronts a Node harness, and that is most of what makes this one different.
Hermes has no base-URL variable to move. `OPENAI_BASE_URL` is read in exactly one place in the
shipped source -- an advisory warning that fires when it is set and the provider is not `custom` --
and the origin itself lives in `config.yaml` under `model.base_url`, which is what the official
configuration reference describes. That was established against a listener rather than inferred:
with `model.base_url` pointed at a dead port and `OPENAI_BASE_URL` at a live one, the run dies on
the dead port and the listener records nothing, `--provider custom` included. orca does not rewrite
a user's global config, so the adapter redirects nothing.
What works is the transport. Hermes honours `HTTPS_PROXY` all the way down -- through a proxy
logging CONNECT targets, one question sends two of them to the model origin, the catalogue read and
the completion -- so `--tls-intercept` reaches it.
The trust store needs nothing added, which is worth recording because it looks as though it would.
The catalogue goes out through `requests` and the completion through httpx under the OpenAI SDK,
and httpx pins certifi. But Hermes resolves that client's CA itself, and the chain in
`agent/ssl_verify.py` reads `HERMES_CA_BUNDLE`, then `SSL_CERT_FILE`, then `REQUESTS_CA_BUNDLE`,
then `CURL_CA_BUNDLE` -- orca already sets the second. An earlier draft of this branch added the
vendor variable to `setupTlsCapture` on the theory that it was needed; removing it again and
re-recording produced the same single model exchange, status 200, 10,615 input tokens, so it is
not here.
`inference-api.nousresearch.com` joins `DEFAULT_TLS_HOSTS` so Hermes on Nous' own models needs no
extra `--tls-hosts`. Deliberately not joining it: `portal.nousresearch.com`, which is billing and
subscription management and where the API key is issued -- the same line this list draws at every
other vendor's console.
Verified against Hermes Agent v0.21.0 (2026.8.31) by running it, on the `opencode-free` tier that
is served anonymously so no credential enters the recording:
orca record hermes --tls-intercept --tls-hosts '+opencode.ai' -- \
--provider opencode-free -m nemotron-3.5-lightning-free -z 'Compute 212000 plus 8 ...'
One model exchange: 42,305 bytes, 19 tools, a 7,742-character system prompt, answered streamed
with status 200 for 10,615 input and 88 output tokens -- and `orca replay` reused 3/3. `opencode.ai`
is named explicitly rather than defaulted in, being a website as much as a gateway.
The prompt is filed under `prompt/HERMES/`, which puts the same free model in the table twice: 9,655
characters and 11 tools from OpenCode against 7,742 and 19 from Hermes. Same model, same anonymous
endpoint -- the harness is the only variable, and that is the pair that isolates it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
5483933 to
cd35c7e
Compare
The previous commit records an exchange the agent walked away from, and files it with `status 0` because no response header ever arrived. On the timeline that renders as `status 0` and nothing else, which reads like a failure the run never had -- the agent left on purpose, having got what it asked for. So the row says `status 0 · client left`. Tone stays normal: leaving early is the agent's own behaviour, not something orca did wrong, and marking it attention would train the eye to ignore the marker on runs where every turn ends this way. A Hermes run has two of these per question. Mutation-checked: dropping the marker turns exactly this test red. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…answer
Follow-on to the abandoned-exchange fix, and found by forking a real recording rather than by
reading the code. `warnUnclaimed` fires for any decrypted POST with a JSON body that reaches
`onNetExchange`, and since that commit a call the agent abandoned before the origin answered lands
there too — deliberately, because an exchange with no response cannot be a model exchange whatever
dialect exists.
The warning said the opposite:
warn tls.unclaimed_path path=/zen/v1/chat/completions
detail="decrypted and recorded, but no wire dialect claims this path"
next="docs/plugins.md to add a dialect for it"
That path is claimed. The openai dialect had already recognised the same path in the same run, one
event earlier. Writing a plugin would change nothing, because there is no response to parse — and
Hermes produces two of these per question, so the advice would print on every turn of every run.
So the warning skips a `status === 0` exchange, which is exactly the condition that made the
promotion skip it. A response that did arrive on an unrecognised path still warns; the existing
test for that is unchanged and still passes.
Testing it needed two small things in the fixtures: an origin that never answers, so the state can
be reached without racing a fast one, and a `leave` mode in the shared proxy caller for a client
that stops after sending its request. Mutation-checked: removing the guard turns the new test red
and nothing else.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
⚠️ Outdated review — the PR head changed from cd35c7e to 2bb5ef1 during the review; these findings describe a diff the PR no longer has.
These findings describecd35c7e4abc2. Re-run the review to check the PR as it stands now.
🐳 OrcaCode Review
✅ No findings — nothing to flag in this PR. Great work!
OrcaCode Review — Route Smarter. Ship Safer. Spend Less.
Engine-reported: 314 calls · 18.9M tokens · 98% cached
❤️ Share · Install OrcaCode Review
Free on GitHub — the review runs on your own OrcaRouter key. If it helped, a shout-out goes a long way.
Share: X · Reddit · LinkedIn
Follow: X · Discord · LinkedIn · OrcaRouter
There was a problem hiding this comment.
🐳 OrcaCode Review
Found 1 issue in this PR: 🟠 1 P1.
OrcaCode Review — Route Smarter. Ship Safer. Spend Less.
Engine-reported: 330 calls · 18.3M tokens · 98% cached
❤️ Share · Install OrcaCode Review
Free on GitHub — the review runs on your own OrcaRouter key. If it helped, a shout-out goes a long way.
Share: X · Reddit · LinkedIn
Follow: X · Discord · LinkedIn · OrcaRouter
| } | ||
| }); | ||
| originRes.on('error', reject); | ||
| originRes.on('aborted', abandon); |
There was a problem hiding this comment.
🟠 P1 HTTP/1.1: do not treat an origin-side mid-response abort as client abandonment
New in this commit (the old code had no 'aborted' listener and rejected on originRes 'error', so the server's catch wrote a 502). When the ORIGIN connection dies after response headers but before the body completes (RST or premature FIN — reproduced on Node 22: the proxy's originRes emits 'aborted' first, then 'error' ECONNRESET), this fires abandon(), which records the exchange and calls resolve() — but the downstream client is still connected and waiting. abandon() never writes to res; the later originRes 'error' handler's reject(err) is a no-op because the promise already resolved, so the server-level catch that would have sent a 502 never runs. Verified with a plain-HTTP simulation of this exact wiring: after "originRes aborted", no 502 is ever sent and the client waits indefinitely (in the simulation it was still waiting after 8 s; real agents' SSE reads typically have no or very long timeouts). Before this commit the same origin death produced a prompt 502. The net effect: any origin crash/network break mid-stream on the HTTP/1.1 interception path silently hangs the agent's in-flight call, and files a trace record claiming "abandoned: client left" (the client never left). This is precisely the "no 502 for the client to see" failure the new upstream-error handler comments warn against, re-introduced on the response object. Fix: on 'aborted' only abandon when the downstream client is actually gone, otherwise reject so the caller's 502 path runs: originRes.on('aborted', () => { if (res.destroyed) abandon(); else reject(new Error('origin aborted mid-response')); });.
1 participant
Orca-Code-Review — push 1
❌ 1 finding blocks merge
Four commits. The first is the one that matters beyond Hermes; the last two came out of running
the result rather than reading it.
The bug
An intercepted streaming response was recorded only if it reached
end. A harness that stopsreading the moment it has its answer never gets there, so the run kept nothing — and then said:
which is the opposite of what happened. orca had decrypted the request, forwarded it, and seen the
response headers. Any harness that abandons an SSE stream early hits this; Hermes does it on every
turn.
Instrumenting the CONNECT handler is what settled it — three connections, all
allowed=true, twoPOST /zen/v1/chat/completionsarriving at the intercepting server with no TLS error, and zeroevents in the trace. Both forwarders now hoist what they have read out of the response handler and
record it once, from whichever path settles first.
NetExchange.abandonedsays which happened:the agent stopped asking for bytes, as against
responseTruncated, where orca stopped keepingthem. The h2 path had the same gap and is fixed the same way.
One follow-on in the same commit:
server.tsstops promoting a response-less call to a modelexchange. A POST abandoned before the origin answered satisfies the dialect check on its path
alone, and promoting it puts an entry in the replay set that can answer nothing and cannot be
forked. On a real Hermes run this is the difference between one clean model exchange plus one
recorded probe and three model exchanges, two of them empty.
Worth a reviewer's eye: the client-gone signal is
res.destroyedand deliberately notreq.destroyed. A request with no body destroys itself once read, so asking aboutreqturnsevery upstream failure — an origin certificate orca refuses to trust, most of all — into a silent
abandonment with no 502 for the client. The repo's existing test for that refusal caught the first
draft.
What running it turned up
The last two commits are things the first one broke or left unsaid, found by recording, replaying
and forking a real run rather than by re-reading the diff.
status 0, andorca showrendered exactly that — indistinguishable from a failure the run never had. The rownow reads
status 0 · client left, and staysnormaltone: the agent leaving early is its ownbehaviour, and marking it attention would train the eye to ignore it on runs where every turn
ends this way.
warnUnclaimedstarted giving advice that could not help. Forking a Hermes recording printedno wire dialect claims this pathfor/zen/v1/chat/completions— a path the openai dialect hadclaimed one event earlier in the same run. It fired because the abandoned probe now reaches
onNetExchangeby design. It now skips astatus === 0exchange, which is the same conditionthat made the promotion skip it; a response that did arrive on an unrecognised path still warns.
The adapter
Hermes is the first Python harness here, and the first with no base-URL variable to move at
all.
OPENAI_BASE_URLappears in one place in the shipped source — an advisory warning for whenit is set and the provider is not
custom— and the origin lives inconfig.yamlundermodel.base_url, which is what the official configurationreference describes. Verified
against a listener rather than inferred: point
model.base_urlat a dead port andOPENAI_BASE_URLat a live one and the run dies on the dead port with the listener untouched,--provider customincluded. orca does not rewrite a user's global config, so the adapterredirects nothing and
--tls-interceptis the route.No CA handling was needed, and an earlier revision of this branch wrongly added some. Hermes
sends the catalogue read through
requestsand the completion through httpx under the OpenAI SDK,and httpx pins certifi — so
SSL_CERT_FILElooks like it should not reach it. It does: Hermesresolves that client's CA itself, and the chain in
agent/ssl_verify.pyreadsHERMES_CA_BUNDLE,then
SSL_CERT_FILE, thenREQUESTS_CA_BUNDLE, thenCURL_CA_BUNDLE. I had addedHERMES_CA_BUNDLEtosetupTlsCapturebelieving it was the missing piece; removing it again andre-recording gave the same single model exchange, status 200, 10,615 input tokens. The variable is
gone from this branch and
attach.tsis untouched. The missing completion had exactly one cause,which is the bug above.
inference-api.nousresearch.comjoinsDEFAULT_TLS_HOSTS.portal.nousresearch.comdeliberatelydoes not — billing, subscription management, and where the API key is issued.
Verification
Hermes Agent v0.21.0 (2026.8.31), installed from git, on the
opencode-freetier that isserved anonymously so no credential enters the recording:
orca replayorca replay --from 5 --model deepseek-v4-flash-freenet.*withabandoned=true,status=0Regression-tested against harnesses that were already working, since this rewrites the forwarder
every transport capture goes through:
record kilo --tls-interceptreused=2/2,unmatched=0, exit 0record opencode --tls-intercepth2 ECONNRESETwarnings it prints are onmaintoorecord generic-openai(plain HTTP, no interception)opencode.aiis passed explicitly rather than defaulted in, being a website as much as a gateway.The prompt lands in
prompt/HERMES/, which puts one free model in the table twice — 9,655 chars/ 11 tools from OpenCode against 7,742 / 19 from Hermes. Same model, same anonymous endpoint; the
harness is the only variable.
Tests
Five new cases, each mutation-checked — reverting the fix it covers turns exactly that test red and
nothing else:
Full suite compared per-test against
main: 23 failures on each, the same 23 — every one apre-existing Windows failure (
0o600file modes, symlinks). No regressions and no differences.🤖 Generated with Claude Code