Correct offline-agent guidance from measurements on a 24 GB M2 - #57
Conversation
Three corrections and two additions, all measured on the machine rather than recalled, while setting it up for offline agentic work. Corrections: - The "2048-token context window" claim is stale. Ollama now picks the default from detected memory (4k/32k/256k), per `ollama serve --help`. A 24 GB Apple-silicon machine gets 4096, not 32k, because only ~75% of unified memory is GPU-addressable. Adds `ollama ps` as the way to check what you actually got, and a Modelfile route for clients that cannot send num_ctx themselves. - qwen2.5-coder does not emit tool calls (0/4 attempts, finish_reason "stop"), so it cannot drive an agentic loop, though it remains a strong completion model. granite4:7b-a1b-h succeeded 3/3 plus a multi-turn round trip at a third of the size. `ollama show` capability metadata is wrong in both directions, so only a real request settles it. - The hardware-tier table overstated 24 GB of unified memory. The 30-32B tier is unreachable there: qwen3-coder's smallest tag is 19 GB against a ~18 GB ceiling, and raising a 14B from 4k to 32k context cost ~5.5 GB of KV cache. Additions: - Ollama serves an Anthropic-compatible /v1/messages endpoint, so Claude Code runs against a local model with no proxy. - Harness overhead bounds what a small model can do, with the tool surface added as a sixth guardrail. Also documents edit_format: whole for small models: with `diff`, granite4 fixed the target function and silently deleted an adjacent one in 3/3 runs, committing a message that mentioned only the fix. Closes #56
api.anthropic.com returns 404 to a plain GET, so lychee rejected it. The probe only needs any external host to demonstrate the block, and a vendor-neutral one states the point better: what matters for an offline claim is that no outbound traffic succeeds. Verified all six external URLs this branch adds now return 200.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
All four confirmed, none rebutted. 1. Restore the "Falling Back Between Cloud and Local Automatically" heading, which my edit dropped, orphaning the LiteLLM content under the new Claude Code section. Found by reading the diff's deleted lines, which I should have done before the first push. 2. Drop "a third of the size". It was true only of resident size at 32k (4.7 vs 15 GB) and false by parameter count, with no measure stated. Now says "half the parameter count", which is unambiguous. 3. Correct the tool-calling claim, which was wrong. Re-checking with the full `ollama show` output shows gemma4:12b DOES advertise `tools`; my original reading came from `grep -A3`, which truncated the capability list one line before `tools`. The "wrong in both directions" claim is withdrawn. Re-measuring also gave a sharper result than the one it replaces. qwen2.5-coder does not ignore the tools: it emits a correct tool call as prose in `content` while leaving `tool_calls` empty, so a harness sees a plain reply and never runs the tool. The rule is that an advertised capability is necessary but not sufficient. 4. Add HTTP-status checking to the tool-calling snippet. As written it could not distinguish a model declining to call a tool from a request that errored, which is the ambiguity the same file warns about elsewhere. The no-call branch now prints `content`, which is what makes the qwen failure mode visible. Verified against all three branches: granite4 (agent), qwen (no call), bad tag (HTTP 404).
ARD round 1All four findings Addressed, none rebutted. The review was right on every point, including the one I would have defended.
Finding 3 deserves the detail, because the reviewer inferred the right conclusion from the wrong reason. The suggestion was that I had mixed this up with So the "wrong in both directions" claim is withdrawn. All three models I tested advertise Re-measuring with full output produced a sharper finding than the one it replaces, so the section is stronger for the correction. {"name": "read_file", "arguments": {"path": "src/main.py"}}Right JSON, wrong field. A harness reads Finding 4 was the same class of mistake one level down, and the file criticises it elsewhere in its own words: a check whose failure path and pass path look identical. The snippet now separates the status code, and prints Finding 1 was mine and was avoidable: I dropped the heading while inserting a section, and did not read the diff's deleted lines before pushing. I ran that check this round and it is clean apart from intentional replacements. Finding 2: "a third" was true only of resident size at 32k (4.7 vs 15 GB) and false by parameter count, with no measure stated. Now "half the parameter count". Re-verified after the changes: render clean with zero warnings, 0 banned-punctuation hits in 402 added lines with a positive control confirming the detector fires, spellcheck differenced against |
|
Claude finished review — View run All four issues from the previous review round have been fixed in commit
I also checked the fix commit itself for new problems and found none: no re-introduction of unicode em/en-dashes (the repo's banned-punctuation check), no new 3+-item lists rendered as prose instead of bullets, no new raw HTML, and the No new issues found in this round. VerdictReady for merge |
|
Closes #56.
Everything here was measured on the machine while setting it up for offline agentic work, not recalled. Each claim states the command that produced it so a reader can re-measure rather than inherit.
Corrections
The 2048-token context claim was stale. Ollama now derives the default from detected memory, per its own
ollama serve --help:A 24 GB Apple-silicon machine lands in the 4096 tier, not the 32k its headline number suggests, because only ~75% of unified memory is GPU-addressable. The chapter now says how to check (
ollama psprints the live context) instead of asserting a number, and gives three ways to raise it, distinguished by which clients each one reaches. The Modelfile route is the only one that reaches clients that cannot sendnum_ctxthemselves.qwen2.5-codercannot drive an agentic loop. It was recommended throughout and listed first in the agentic-model table. Against a single-function tool schema it returnedfinish_reason: stopwith no tool call, 0/4.granite4:7b-a1b-hreturned a well-formed call 3/3 plus a full multi-turn round trip, at a third of the size. It remains a good completion model, and the text now says so rather than dropping it.Relatedly,
ollama showcapability metadata is unreliable in both directions:qwen2.5-coder:14badvertisestoolsand does not call them;gemma4:12badvertises none and calls them correctly. Only a real request settles it, so the chapter gives a one-linecurlto run.The hardware-tier table overstated 24 GB of unified memory. The 30-32B tier is not reachable there:
qwen3-coders smallest tag is 19 GB against a ~18 GB addressable ceiling, with no smaller variant, and context is charged on top -- raising a 14B from 4k to 32k moved it from 9.5 GB to 15 GB resident.Additions
Claude Code offline. Ollama serves an Anthropic-compatible
/v1/messagesendpoint, so any Anthropic-API client can be pointed at a local model with no proxy. Verified returning a real Anthropic-shaped response.Harness overhead, added as a sixth guardrail. A model that tool-calls correctly against the raw API can still fail inside a heavyweight harness, because the system prompt and tool schemas crowd out the task. Asked to read one file, granite4 under Claude Code produced a paragraph about missing credentials with a GitHub MCP server loaded; invented a task list copied from a tool description with MCP disabled; and still ignored the prompt with the surface cut to Read/Grep/Glob. The same model through
aiderfixed a real bug and committed in 17 seconds.edit_format: wholefor small models, with the most important finding in the PR:diffwholeThe
difffailure is not a refused edit. In all three runs the model fixed the target function and silently deleted the adjacent one, then committed a message naming only the intended fix. Nothing in the commit message or exit status mentioned the deletion. That is the compounding-error hazard in its most concrete form, and it is also a case of the chapter own guardrail working: committing every step meant git still had the original.Verification
quarto render chapters/coding-agents.qmd-- clean, zero warnings. Two link warnings surfaced mid-work and were fixed:mcp-server-setup.qmdresolves relative to the including file, and that chapter appears later anyway, so a cross-reference would have been a forward reference. The acronym is expanded inline instead.mainso only genuinely new words counted --Modelfile,Ollama's,schemas, added toinst/WORDLIST. Re-run is clean.Note on scope
references.bibis untouched; no new citations were added. The companion issues (ai-config#1292, gha#436) still carry the runnable halves -- this PR only corrects the explanatory side, per the repo division of labor.