fix: compat issues #57/#58/#59 + Liquid AI tool-call parser (v1.3.1)

.gitignore

@@ -80,3 +80,15 @@ marrow.toml
 !marrow/features_1_6.marrow
 !marrow/quality_monitor.marrow
 !marrow/read_guard.marrow
+
+# Kiro IDE (agent workspace, specs, hooks, and scratch)
+.kiro/
+.kiro-tmp/
+
+# Tooling / agent scratch state
+.qartez/
+.compare-modified.json
+
+# Harbor benchmark job output (generated per run, large)
+jobs/
+bench/harbor/

.npmignore

@@ -13,6 +13,15 @@ node_modules/
 jobs/
 .compare-modified.json
 
+# Agent/IDE scratch (not part of the published package)
+.kiro/
+.kiro-tmp/
+.qartez/
+.github/
+
+# Tests are not needed by consumers of the package
+test/
+
 # Prebuilt binary tarballs (shipped via GitHub releases, not npm)
 tmp*/
 *.tar.gz
@@ -47,3 +56,7 @@ marrow/
 marrow.toml
 core.marrow
 smallcode.marrow
+
+# TypeScript sources in src/compiled/ — only the compiled .js ships at runtime.
+# build.js generates the .js from these; consumers never need the .ts.
+src/compiled/**/*.ts

CHANGELOG.md

@@ -1,5 +1,85 @@
 # Changelog
 
+## [1.3.1] - 2026-05-29
+
+### fix: compatibility issues #57, #58, #59
+
+Three reported environment-compatibility bugs:
+
+- **#58 (zsh shell)** — `src/tools/shell_session.js` honoured `$SHELL`
+  when it matched `bash|zsh` but always passed bash-only flags
+  (`--norc --noprofile -i`). zsh rejects `--norc` and exits immediately,
+  so every `bash` tool call failed with "shell exited" for zsh users.
+  Now always launches `bash` on POSIX (the sentinel command-wrapper
+  emits bash/POSIX syntax anyway). `cmd.exe` on Windows unchanged.
+- **#58 (stdin EPIPE crash)** — when the shell died mid-write, the async
+  `'error'` (EPIPE) event on `proc.stdin` had no listener, so Node
+  escalated it to an uncaught exception and crashed the whole process.
+  Added a stdin `'error'` handler that marks the shell dead and lets the
+  next `run()` auto-restart it.
+- **#58 (web tools not advertised)** — `buildSystemPrompt` (both
+  `bin/model_client.js` and `bin/smallcode.js`) enumerated tools in prose
+  but never mentioned `web_search` / `web_fetch`, even with
+  `SMALLCODE_WEB_BROWSE=true`. Small models trusted the prose over the
+  raw `tools` array and refused research tasks. Now conditionally
+  describes the web tools when browsing is enabled.
+- **#59 (remote Ollama)** — the Ollama health check ignored
+  `SMALLCODE_BASE_URL` and always probed `OLLAMA_HOST`/localhost, so a
+  remote Ollama server reported "Ollama not running" with no diagnostics.
+  Now probes the configured base URL (stripping a trailing `/v1`), carries
+  auth, converts URL-embedded basic-auth (`https://user:pass@host`) into a
+  proper `Authorization: Basic` header, and surfaces the real error
+  (ECONNREFUSED / ENOTFOUND / TLS) instead of a generic message.
+- **#57 (prebuild-install deprecation)** — documented in the README that
+  the `prebuild-install@7.1.3: No longer maintained` warning is a harmless
+  upstream deprecation in a transitive optional dependency
+  (`budget-aware-mcp` → `better-sqlite3` → `prebuild-install`) with no
+  newer version to bump to. Added `--omit=optional` guidance.
+
+Also fixed a latent race in `shell_session.js` surfaced while testing:
+late `exit`/`data` events from a superseded shell process could stomp a
+freshly-spawned one. Event handlers are now scoped to their specific
+child process.
+
+Test coverage: `test/shell_session.test.js` (4 cases), `test/web_prompt.test.js`
+(3 cases), and 4 new basic-auth cases in `test/provider_compat.test.js`.
+Full suite: 148 passing.
+
+### fix: Liquid AI tool-call parser (lfm2.x compatibility)
+
+Adds support for Liquid AI's tool-call format. `lfm2.5-8b-a1b-apex` and
+related models emit calls as Python keyword-arg syntax wrapped in
+`<|tool_call_start|>[func(kw='val')]<|tool_call_end|>` markers. LM Studio
+passes this through unchanged because it has no parser for Liquid's
+chat template, so SmallCode previously saw zero tool calls and the model
+appeared completely broken.
+
+- New: `src/tools/liquid_tool_parser.js` — Python-literal-aware parser
+  for the markered format. Handles single/double-quoted strings with
+  `\n`, `\t`, `\\`, `\'`, `\xNN`, `\uNNNN` escapes; numbers, booleans
+  (`True`/`False`/`None`); nested lists and dicts.
+- `src/tools/tool_call_extractor.js` calls the new parser as its
+  highest-priority recovery path. Also falls back to scanning
+  `message.reasoning_content` when `message.content` is empty (Liquid AI
+  splits visible output and chain-of-thought into separate fields).
+- Test coverage: `test/liquid_tool_parser.test.js` (13 cases incl.
+  multi-call lists, JSON-in-content, and reasoning-content fallback).
+
+### bench: polyglot-mini comparison — lfm2.5-8b-a1b-apex vs gemma 4 e4b
+
+Ran the polyglot-mini suite (19 tasks) back-to-back against the LM Studio
+host at `10.0.0.20:1234`. Results in `bench/results/polyglot-mini.md`,
+raw JSON under `.smallcode/benchmarks/`.
+
+- `huihui-gemma-4-e4b-it-abliterated` — **16/19 (84%)**
+- `lfm2.5-8b-a1b-apex` (with parser fix) — **9/19 (47%)**, up from 1/19 (5%)
+
+Remaining lfm2.5 failures are dominated by `finish_reason='length'`
+truncation when the model exhausts its budget on `reasoning_content`
+before producing visible output. Documented as a known limitation in
+`bench/results/polyglot-mini.md`; not addressed in this pass to keep
+the change scoped to the format-parsing fix.
+
 ## [1.3.0] - 2026-05-26
 
 ### feat: plugin system core + provider wizard (#28, #29)

README.md

@@ -64,6 +64,16 @@ SmallCode includes [BoneScript](https://github.com/Doorman11991/BoneScript) and
   - **Windows**: Visual Studio Build Tools with "Desktop development with C++" workload, or `npm install -g windows-build-tools`
 - **If build fails, SmallCode still works** — it falls back to JSON-based memory automatically
 
+> **Note on the `prebuild-install@7.1.3: No longer maintained` warning** (issue #57):
+> this is a harmless upstream deprecation notice from a transitive dependency
+> (`budget-aware-mcp` → `better-sqlite3` → `prebuild-install`). `prebuild-install`
+> has no newer published version, and `better-sqlite3` still depends on it, so the
+> warning cannot be silenced by a version bump. It does **not** affect the install —
+> npm warnings are advisory and the package installs normally. If you prefer to skip
+> the native dependency entirely, install without optional deps:
+> `npm install -g smallcode --omit=optional` (you lose FTS5 memory search but keep the
+> JSON memory fallback).
+
 ### Configuration
 
 Create a `.env` file in your project root:
@@ -182,7 +192,7 @@ Halves the schema context overhead. Model picks a category (read/write/search/ru
 Detects repetition loops, patch spirals (stuck on corrupted file → forces rewrite), and greeting regression (model lost context → re-injects task). Saves tokens and time.
 
 ### Forgiving Tool Call Parser
-Small models produce messy output. SmallCode parses tool calls from JSON, YAML, XML, Hermes format, or plain text. Auto-repairs common mistakes (wrong param names, type mismatches).
+Small models produce messy output. SmallCode parses tool calls from JSON, YAML, XML, Hermes format, Liquid AI's `<|tool_call_start|>[func(kw='val')]<|tool_call_end|>` markers (lfm2.x), or plain text. Auto-repairs common mistakes (wrong param names, type mismatches). Falls back to scanning `reasoning_content` when `content` is empty (LM Studio reasoning models).
 
 ### Patch-First Editing
 Search-and-replace as the primary edit primitive. Small models can't reliably reproduce entire files — they truncate, hallucinate, or drift. `patch` is safer and more context-efficient.

bench/results/polyglot-mini.md

@@ -4,22 +4,108 @@
 workspace — no shared state, no internet, no installed toolchain required for
 the verify step (all checks are file-content based).
 
-## Latest result
+## Latest results
 
-| Model | Pass rate | Mean time/task | Date |
-|---|---|---|---|
-| huihui-gemma-4-e4b-it-abliterated (8B) | **19/19 (100%)** | 11.3s | 2026-05-21 |
+| Model | Pass rate | Mean time/task | Date | Notes |
+|---|---|---|---|---|
+| huihui-gemma-4-e4b-it-abliterated (8B) | **16/19 (84%)** | 24.2s | 2026-05-29 | gemma run |
+| lfm2.5-8b-a1b-apex (8B / A1B MoE)      | **9/19 (47%)**   | 20.0s | 2026-05-29 | with Liquid AI parser |
+| lfm2.5-8b-a1b-apex (8B / A1B MoE)      | 1/19 (5%)        | 41.5s | 2026-05-29 | before parser fix |
+| huihui-gemma-4-e4b-it-abliterated (8B) | 19/19 (100%)     | 11.3s | 2026-05-21 | original baseline |
 
-## Per-language breakdown
+The earlier 19/19 gemma run is preserved for historical reference. The
+2026-05-29 re-run was performed back-to-back with the lfm2.5 runs on the
+same LM Studio host so the two columns are directly comparable.
 
-| Language | Tasks | Passed |
-|---|---|---|
-| Python | 5 | 5 |
-| JavaScript | 4 | 4 |
-| TypeScript | 3 | 3 |
-| Shell | 3 | 3 |
-| Markdown | 2 | 2 |
-| JSON | 2 | 2 |
+## Per-language breakdown — 2026-05-29
+
+| Language   | Tasks | gemma 4 e4b | lfm2.5 (post-fix) | lfm2.5 (pre-fix) |
+|---         |---    |---          |---                |---               |
+| Python     | 5     | 5/5         | 3/5               | 0/5              |
+| JavaScript | 4     | 3/4         | 1/4               | 1/4              |
+| TypeScript | 3     | 3/3         | 2/3               | 0/3              |
+| Shell      | 3     | 2/3         | 0/3               | 0/3              |
+| Markdown   | 2     | 2/2         | 2/2               | 0/2              |
+| JSON       | 2     | 1/2         | 1/2               | 0/2              |
+
+## Why lfm2.5-8b-a1b-apex was failing — investigation
+
+The original 1/19 result was caused by a **tool-call format mismatch**, not
+model quality. Three issues compound:
+
+### 1. Liquid AI's tool-call template (root cause, fixed)
+
+`lfm2.5-8b-a1b-apex` emits tool calls using Python keyword-arg syntax
+wrapped in markers that LM Studio passes through to `message.content`
+verbatim because it does not have a parser for Liquid's chat template:
+
+```
+<|tool_call_start|>[write_file(path='hello.py', content='def greet(name):\n    return f"Hello, {name}!"')]<|tool_call_end|>
+```
+
+SmallCode's existing `tool_call_extractor` only knew about
+`<tool_call>{json}</tool_call>`, fenced-JSON, and bare-leading-JSON
+formats. None matched, so 17 of 19 tasks recorded zero tool calls.
+
+**Fix:** Added `src/tools/liquid_tool_parser.js` — a Python-literal-aware
+parser that handles single/double-quoted strings, escape sequences
+(`\n`, `\t`, `\\`, `\'`), numbers, booleans (`True`/`False`/`None`),
+nested lists, and dicts. Wired into `tool_call_extractor.js` as the
+highest-priority recovery path.
+
+### 2. Empty `content`, populated `reasoning_content` (partially fixed)
+
+LM Studio surfaces lfm2.5's chain-of-thought as a separate
+`reasoning_content` field on the response. When the model's
+`max_tokens` budget is exhausted by reasoning, `content` arrives empty
+even though the model intended to call a tool.
+
+**Fix:** When `message.content` is empty, the extractor now also scans
+`message.reasoning_content` for Liquid-format tool calls. This recovers
+the cases where the call landed in reasoning instead of content.
+
+### 3. `finish_reason='length'` truncation (not yet fixed)
+
+The model frequently spends 200-2400 reasoning tokens on a single turn,
+and the first call in a multi-turn task often returns
+`finish_reason='length'` with empty content. SmallCode's quality monitor
+warns (`empty_response`) but doesn't retry with a higher cap or with
+thinking disabled.
+
+This is the dominant remaining failure mode. About half of the post-fix
+failures show this pattern: the model's first 1-2 calls hit `length`,
+then it produces a successful Liquid call, but by then the agent loop
+has already been derailed (sometimes infinitely loops on `bash` heredoc
+calls that fail on Windows with exit code 9009).
+
+A follow-up fix would catch `finish_reason='length' && content=='' && tool_calls.length===0`
+in the agent loop and either retry with `enable_thinking=false` or with
+a doubled `max_tokens`. Not implemented in this pass to keep the change
+scoped.
+
+## Per-task — 2026-05-29 (post-fix vs gemma)
+
+| Task              | gemma 4 e4b      | lfm2.5 post-fix    |
+|---                |---               |---                 |
+| py-fibonacci      | ✅ 20.9s · 2t    | ✅ 9.1s · 2t       |
+| py-class-account  | ✅ 205.2s · 22t  | ❌ 9.8s · 2t       |
+| py-fix-list       | ✅ 15.2s · 4t    | ❌ 3.9s · 0t       |
+| py-add-test       | ✅ 8.9s · 3t     | ✅ 10.2s · 2t      |
+| js-double         | ✅ 4.9s · 2t     | ✅ 6.5s · 2t       |
+| js-arrow          | ✅ 13.2s · 3t    | ❌ 6.2s · 1t       |
+| js-package        | ✅ 13.6s · 4t    | ❌ 11.4s · 0t      |
+| js-fix-async      | ❌ 10.3s · 2t    | ❌ 167.5s · 0t     |
+| ts-interface      | ✅ 3.7s · 1t     | ✅ 4.6s · 1t       |
+| ts-generic        | ✅ 8.4s · 2t     | ❌ 6.5s · 2t       |
+| ts-tsconfig       | ✅ 3.9s · 1t     | ✅ 4.9s · 1t       |
+| sh-list           | ✅ 10.3s · 2t    | ❌ 78.7s · 0t      |
+| sh-makefile       | ❌ 49.9s · 12t   | ❌ 12.4s · 0t      |
+| sh-script         | ✅ 52.1s · 10t   | ❌ 11.4s · 1t      |
+| md-readme         | ✅ 11.9s · 2t    | ✅ 7.6s · 1t       |
+| md-api            | ✅ 9.8s · 1t     | ✅ 6.6s · 1t       |
+| json-config       | ✅ 3.7s · 1t     | ✅ 5.7s · 1t       |
+| json-fix          | ❌ 4.2s · 2t     | ❌ 7.3s · 1t       |
+| multi-imports     | ✅ 9.5s · 2t     | ✅ 10.1s · 2t      |
 
 ## Setup
 
@@ -28,13 +114,21 @@ Requires a running OpenAI-compatible endpoint. Set `SMALLCODE_BASE_URL` and
 
 ```bash
 npm run bench:polyglot
+# or compare two models directly:
+node bench/harness.js --suite polyglot-mini --model lfm2.5-8b-a1b-apex --base-url http://10.0.0.20:1234/v1
+node bench/harness.js --suite polyglot-mini --model huihui-gemma-4-e4b-it-abliterated --base-url http://10.0.0.20:1234/v1
 ```
 
 Results are saved to `.smallcode/benchmarks/<run-id>.json`.
 
 ## Notes
 
 - Run on Windows with LM Studio serving the model locally at `10.0.0.20:1234`
-- Timeout per task: 120s
-- The model used is an 8B parameter model — larger models will generally score
-  higher and run faster
+- Timeout per task: 240s
+- lfm2.5-8b-a1b-apex is an A1B MoE — 1B active parameters per token out
+  of an 8B total, served as an apex-quality quant. The 1B active count
+  (not the quantisation) is the architectural constraint here. It splits
+  reasoning and visible output into separate `reasoning_content` and
+  `content` fields, which makes it sensitive to `max_tokens` budgets and
+  to harnesses that don't recognise Liquid AI's Python-kwarg tool-call
+  syntax.

bin/config.js

@@ -321,22 +321,52 @@ async function checkEndpoint(config) {
     }
   }
 
-  // Ollama endpoint
-  const host = process.env.OLLAMA_HOST || 'http://localhost:11434';
+  // Ollama endpoint.
+  // Prefer the user-supplied base URL (SMALLCODE_BASE_URL) over OLLAMA_HOST /
+  // localhost so a remote Ollama server is actually probed. Previously this
+  // branch always hit localhost and reported "Ollama not running" even when
+  // the user pointed SMALLCODE_BASE_URL at a remote host (issue #59).
+  // Ollama's native API lives at the host root (/api/tags), so strip any
+  // trailing OpenAI-style /v1 and normalise the trailing slash.
+  let host = config.model.baseUrl || process.env.OLLAMA_HOST || 'http://localhost:11434';
+  host = host.replace(/\/v1\/?$/, '').replace(/\/+$/, '');
+  const tagsUrl = `${host}/api/tags`;
   try {
-    const response = await fetch(`${host}/api/tags`);
-    if (!response.ok) return false;
+    // Carry auth (e.g. credentials embedded in the URL, or a bearer token /
+    // basic-auth header from buildAuthHeaders) so reverse-proxied Ollama
+    // servers behind auth are reachable.
+    const headers = buildAuthHeaders(config);
+    const response = await fetch(tagsUrl, { headers });
+    if (!response.ok) {
+      console.log(`  ⚠ Ollama returned HTTP ${response.status} from ${tagsUrl}`);
+      if (response.status === 401 || response.status === 403) {
+        console.log(`  Auth required — include credentials in SMALLCODE_BASE_URL or set OPENAI_API_KEY.`);
+      } else if (response.status === 404) {
+        console.log(`  Tip: ${host} has no /api/tags route. Is this actually an Ollama server? For OpenAI-compatible servers set SMALLCODE_PROVIDER=openai.`);
+      } else {
+        console.log(`  Check that Ollama is running and reachable at ${host}.`);
+      }
+      return false;
+    }
     const data = await response.json();
     const models = data.models || [];
     const hasModel = models.some(m => m.name.includes(config.model.name.split(':')[0]));
     if (!hasModel) {
-      console.log(`  ⚠ Model "${config.model.name}" not found in Ollama.`);
+      console.log(`  ⚠ Model "${config.model.name}" not found on the Ollama server at ${host}.`);
       console.log(`  Run: ollama pull ${config.model.name}`);
       return false;
     }
     return true;
-  } catch {
-    console.log('  ⚠ Ollama not running. Start it with: ollama serve');
+  } catch (e) {
+    // Surface the underlying error instead of assuming Ollama is simply not
+    // started — the user may have a wrong URL, DNS failure, or TLS issue.
+    const msg = e && e.message ? e.message : String(e);
+    console.log(`  ⚠ Cannot reach Ollama at ${tagsUrl}: ${msg}`);
+    const hint = /ECONNREFUSED/.test(msg) ? '  If this is a local server, start it with: ollama serve' :
+                 /ENOTFOUND|EAI_AGAIN/.test(msg) ? '  Check the hostname in SMALLCODE_BASE_URL.' :
+                 /certificate|TLS|SSL/i.test(msg) ? '  TLS error — check the https certificate or use http.' :
+                 '  Check that SMALLCODE_BASE_URL points at a running Ollama server.';
+    console.log(hint);
     return false;
   }
 }
@@ -375,6 +405,19 @@ function buildAuthHeaders(config) {
   if (apiKey) {
     headers['Authorization'] = `Bearer ${apiKey}`;
   }
+  // Basic auth embedded in the URL (https://user:pass@host). The fetch() API
+  // does NOT apply URL userinfo automatically, so a reverse-proxied server
+  // behind basic auth would 401. Convert it to an explicit header. An
+  // explicit API key (Bearer, above) takes precedence if both are present.
+  if (!apiKey && modelConfig.baseUrl) {
+    try {
+      const u = new URL(modelConfig.baseUrl);
+      if (u.username) {
+        const creds = Buffer.from(`${decodeURIComponent(u.username)}:${decodeURIComponent(u.password)}`).toString('base64');
+        headers['Authorization'] = `Basic ${creds}`;
+      }
+    } catch { /* not a parseable URL — ignore */ }
+  }
   if (baseUrl.includes('openrouter.ai')) {
     headers['HTTP-Referer'] = 'https://github.com/Doorman11991/smallcode';
     headers['X-Title'] = 'SmallCode';