Skip to content

feat: add Hermes tool call parser#1596

Open
scubamount wants to merge 6 commits into
jundot:mainfrom
scubamount:feat/hermes-tool-call-parser
Open

feat: add Hermes tool call parser#1596
scubamount wants to merge 6 commits into
jundot:mainfrom
scubamount:feat/hermes-tool-call-parser

Conversation

@scubamount
Copy link
Copy Markdown
Contributor

@scubamount scubamount commented Jun 2, 2026

Summary

Adds fallback support for the Hermes-style/Liquid tool call wire format:

<|tool_call_start|>[function_name(arg=value)]<|tool_call_end|>

Some clients/agents emit tool calls using these markers instead of oMLX's currently supported XML/prose bracket formats. Without this fallback, oMLX can surface raw marker text to users instead of returning structured tool calls.

This PR:

  • parses Hermes-style JSON payloads inside <|tool_call_start|>...<|tool_call_end|>
  • parses Python-expression-like bracket payloads such as [read_file(path='/tmp/a.log')]
  • supports multiple calls in one Hermes block via AST parsing, e.g. [tool_a(...), tool_b(...)]
  • suppresses Hermes start/end markers during streaming even when tokenizer metadata is absent
  • suppresses orphan/split Hermes closing markers so <|tool_call_end|> cannot leak if it arrives without a visible open marker
  • strips any remaining complete Hermes envelopes from final assistant text if parsing fails
  • classifies LFM2 text checkpoints as LLMs instead of embeddings by treating lfm2 as ambiguous and relying on architecture/name checks

Verification

/opt/homebrew/opt/python@3.12/libexec/bin/python3 -m pytest tests/test_tool_calling.py -q
161 passed in 0.16s

/opt/homebrew/opt/python@3.12/libexec/bin/python3 -m pytest tests/test_model_discovery.py -q
119 passed in 0.16s

Add support for Hermes-style tool call format:
<|tool_call_start|>[func_name(arg1=val1)]<|tool_call_end|>

Supports both bracket format (Python-style keyword args) and JSON format.
Adds to the fallback chain in parse_tool_calls() and strips markers
when parsing fails to prevent raw markup leaking to API responses.
@scubamount
Copy link
Copy Markdown
Contributor Author

Cross-ref: this PR is the server-side counterpart to NousResearch/hermes-agent#37229 (fix: normalize local tool JSON fallbacks).

These two PRs are co-dependent:

  • omlx#1596 (this PR): parses Hermes-style <|tool_call_start|>...<|tool_call_end|> markers at the inference server layer, so OMLX returns structured tool calls instead of raw marker text.
  • hermes-agent#37229: normalizes local-model assistant JSON with commands/tools_used into OpenAI-style tool calls on the hermes-agent client side.

Without both, local model tool calling is broken:

  • Without omlx#1596 → OMLX leaks raw Hermes markers to hermes-agent
  • Without hermes-agent#37229 → hermes-agent cannot normalize the mixed JSON/bracket payloads that local models emit

Both should merge together for a complete fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant