[native CLI] `needle --serve` hangs in `needle_complete` spin loop on first /complete request

[VIPKaiser]

Summary

Running the native macOS CLI (macos-arm64/needle) in serve mode against the README's quickstart tools file (get_weather + set_thermostat) causes the process to enter a spin loop inside needle_complete on the very first POST /complete request. The connection is never accepted for subsequent requests, no response is written, and the process pegs multiple cores indefinitely.

Environment

• Binary: /Users/tom/Models/cactus-needle2/needle (14,610,568 bytes, matches Cactus-Compute/needle2 macos-arm64/needle byte-for-byte, dated 2026-09-07)
• OS: macOS 27.0 (Darwin 27.0.0), M-series (arm64)
• CLI flags: --serve --port 8082 --tools /tmp/needle-tools.json
• Tool set: the exact get_weather / set_thermostat schemas from the Quickstart section of README.md (see Thermostat example from README truncates with "token budget exhausted" #51 for context)

Reproduction

cd ~/Models/cactus-needle2
./needle --serve --port 8082 --tools /tmp/needle-tools.json &
# logs only the startup line — nothing further is ever written
curl -X POST http://127.0.0.1:8082/complete -d '{"input":"what is the weather in Lagos"}'
# connection times out, process never returns

Observations

1. The log file (needle-server-YYYYMMDD.log) only contains the startup banner — no request lines, so the request never reaches a serialisation path that produces output.
2. lsof -p <pid> shows the listener is bound, but the inbound TCP connection is accepted and then sits with zero bytes read/written until I kill the process. There is exactly one closed stale connection in the FD table after several minutes.
3. RSS stays at ~14.6 MB the entire time — no model weights are being paged in. This is purely a CPU spin, not an inference pass.
4. sample <pid> shows the main thread stuck entirely inside needle_complete:

   1177 Thread_93436044 DispatchQueue_1: com.apple.main-thread (serial)
     1177 start (in dyld)
     1177 needle_complete (in needle) + 4248
     1176 ??? (in needle) load address 0x102988000 + 0xb730
     1176 ??? (in needle) load address 0x102988000 + 0x61e8c
       1176 ??? (in needle) load address 0x102988000 + 0x447c
         781 ??? (in needle) load address 0x102988000 + 0x46944
           781 ??? (in needle) load address 0x102988000 + 0x496b8
             210 ??? (in needle) load address 0x102988000 + 0x5754c
   

   The recursion depth in needle_complete and absence of any symbol resolution confirms this is the grammar-constrained decode loop. 1177/1177 samples are inside it — not a request-handling thread.
5. The bug burns ~380% CPU continuously until the process is killed (verified over a 24h run). No timeout, no log line, no way for the operator to know the request will never complete.

Likely relationship to #51

Issue #51 was a Python-package bug (shared needle_init engine with stale tools) that produced a wrong-but-bounded response — the model reasoned about a description field that wasn't in the schema, then truncated with token budget exhausted. The same get_weather/set_thermostat schema mismatch that produced wrong-tool reasoning in #51 may be putting the native CLI's constrained decode loop into a state with no grammar-accepting continuation, so the engine retries forever instead of falling back to a refusal response like the Python path does.

Expected

Either:

• a structured refuse / truncated JSON response matching the Python package's behaviour (preferred, matches Thermostat example from README truncates with "token budget exhausted" #51's resolution), or
• an iteration-cap that emits an error after N decode steps so the request at least terminates.

Actual

Process spins at ~380% CPU, never logs anything, never writes a response, never returns from needle_complete. Killed only by SIGKILL.

Workaround

None observed. The CLI appears unusable with the README's example tools file. Limiting this to the Python package is not a documented constraint.

[HenryNdubuaku]

Thanks for the detailed trace. I ran the same binary (byte-for-byte the HF needle2 macos-arm64 build) with the quickstart tools, both the hand-written schema and the one @needle.tool generates, and --serve answers the first request in about 20 ms here on macOS 26. Could you attach your exact /tmp/needle-tools.json, and try the Needle 3 build (Cactus-Compute/needle3)? If it still spins on 27 I'll dig in with your file.

[HenryNdubuaku]

I couldn't reproduce this. I pulled the same macos-arm64 binary (14,610,568 bytes, byte for byte), used your two tools and your exact curl, and it answered in a couple of seconds:

{"type":"call",...,"function_calls":[{"name":"get_weather","arguments":{"city":"Lagos"}}]}

Same on the current Needle 3 build. Your RSS staying at 14.6 MB says the weights never got touched, so it hung before inference started.

Two things that would help: what does the tools file look like exactly (a trailing comma or a stray byte would fit the symptom), and how many cores does the machine report? Also worth trying --threads 1 and the current engine, since a lot has changed in the thread pool since 2.0.4.