OCR: GLM-OCR model produces garbage output via MLXVLM

[kiki830621]

Problem

macdoc ocr pipeline runs end-to-end without crashes, but the OCR output is meaningless token repetitions instead of recognized text.

Type

bug

Expected

Running macdoc ocr /tmp/ocr-test.png on an image containing "Hello OCR Test 你好世界" should produce readable Markdown text.

Actual

Output is repeated garbage tokens:

刷刷刷刷刷刷一闪ount刷刷刷刷刷一闪ount刷刷刷刷刷刷...

Or with longer max-tokens:

encyencyencyencyency...指指指指指...Span Span Span...

Context

• Model: EZCon/GLM-OCR-8bit-mlx (8-bit quantized GLM-OCR for MLX)
• Pipeline: OCRPipeline → VLMModelFactory.shared.loadContainer() → MLXLMCommon.generate()
• The MLXVLM library has a GlmOcr.swift implementation, so model architecture should be correct
• Possible causes:
  1. The quantized model (EZCon/GLM-OCR-8bit-mlx) itself may be broken
  2. Image preprocessing (CIImage conversion) may not match what the model expects
  3. Chat template / prompt format may be wrong for this model
  4. Need to test with a known-working model (e.g., mlx-community/Qwen2.5-VL-3B-Instruct-4bit)

Impact

macdoc ocr is unusable until this is resolved — the entire OCR feature depends on correct VLM output.

Next Steps

• Test with mlx-community/Qwen2.5-VL-3B-Instruct-4bit to verify the pipeline works with a different model
• If Qwen works, the issue is GLM-OCR specific (model or config)
• If Qwen also fails, the issue is in OCRPipeline's image/prompt handling

---

Current Status

Phase: diagnosed (round 2)
Last updated: 2026-04-21 by idd-diagnose

Key Decisions

• (2026-04-21) Round 2 refreshed diagnosis posted (issuecomment-4288001978): original 2026-03-28 diagnosis predates fix: PageOCRRunner uses OCRCore.backend API (unblocks main build) #84's backend refactor; issue scope has widened to include general MLX VLM instability (mlx-swift-lm#191 broadcast_shapes upstream bug per MLXBackend.swift docstring).
• (2026-04-21) Strategy restructured into Track A/B/C/D: Track A = investigation (3 reproductions), Track B = GLM-OCR-specific fix, Track C = MLX upstream tracking, Track D = close-path decision depending on A.
• (2026-04-21) Key finding in code: MacDoc+OCR.swift:60-65 already silently routes config.ocrDefaultModel == "glm-ocr" to Qwen3-VL — compatibility shim already in place, effectively side-stepping OCR: GLM-OCR model produces garbage output via MLXVLM #66 for the default path.
• (2026-04-21) Complexity = SDD-warranted (6 open questions: Track A ordering, GLM-OCR deprecation vs fix, MLX-vs-Ollama default, test coverage, pdf-ocr-vs-top-ocr sharing, issue scope). User chose /spectra-discuss.
• (2026-03-28 R1) Original diagnosis recommended Step 1-4: test Qwen2.5-VL first, then isolate GLM-OCR specific vs pipeline. Never actioned.

Scope Changes

• (2026-04-21 R2) Scope widened from "GLM-OCR specific garbage output" to "OCR backend strategy post-fix: PageOCRRunner uses OCRCore.backend API (unblocks main build) #84" — includes general MLX VLM instability risk.
• (2026-04-21 R2) Investigation gate added — Track A 3 reproductions must complete before Track B/C/D can proceed.

Blocking

• Waiting for /spectra-discuss — 6 open questions, primary being "is scope narrow (GLM-OCR only) or broad (MLX VLM path)". Cannot proceed to /spectra-propose without this alignment + Track A evidence.
• Track A requires local Ollama server + multi-GB model downloads; may need separate session with that setup.

Commits (relevant — since issue opened 2026-03-28)

• 1a95969 (fix: PageOCRRunner uses OCRCore.backend API (unblocks main build) #84) wip: swap default OCR model to Qwen3-VL + OCRCore backend abstraction — side-stepped OCR: GLM-OCR model produces garbage output via MLXVLM #66 at default level
• 51e8a07 feat(config): add config ocr subcommand with host profile support — Ollama host management
• f1b975f refactor: simplify PDF pipeline — replace block segmentation with page-level GLM-OCR — pdf ocr still defaults to GLM-OCR
• 692dc17 (refactor: extract pdf-to-latex-swift + ocr-swift to PsychQuant repos; commit Tests/WordToMDTests fixtures (continues #78 pattern) #79) ocr-swift package extracted from macdoc — code now lives in PsychQuant/ocr-swift

Related Issues

• #68 (CLOSED) — hf xet download produced corrupted safetensors (referenced in original diagnosis Step 4)
• #78 — note packages extraction (unrelated, same era)
• #79 — ocr-swift extraction (relocated the code)
• Upstream: ml-explore/mlx-swift-lm#191 — broadcast_shapes crash in VLM inference (actively affects OCR: GLM-OCR model produces garbage output via MLXVLM #66)

[kiki830621]

Diagnosis

Type

bug

Root Cause

Two possible causes (need to isolate):

1. Model-specific issue: `EZCon/GLM-OCR-8bit-mlx` is a third-party 8-bit quantization — not from `mlx-community`. The quantization may have degraded quality. MLXVLM has no pre-registered GLM-OCR model in its `VLMRegistry`, meaning this model hasn't been validated by the Apple team.

2. Pipeline issue unlikely but possible: `OCRPipeline` uses `CIImage(cgImage:)` which is supported by `UserInput.Image`. The prompt "OCR the text in this image." may not match what the model's chat template expects. The `llm-tool` reference uses `ChatSession` for multi-turn, but single-turn should also work.

Evidence

• MLXVLM's `GlmOcr.swift` (1262 lines) has full architecture support including Conv3D patch embed, 2D RoPE, spatial merge via Conv2D downsample, and the GlmOcr-specific merger
• The `Glm46VProcessor` handles image preprocessing internally — no manual image prep needed
• The garbage output pattern (repeated tokens) suggests either weight corruption or wrong tokenizer/chat template pairing

Strategy

• Step 1: Test with a known-working model to isolate pipeline vs model issue:
  ```bash
  hf download mlx-community/Qwen2.5-VL-3B-Instruct-4bit
  macdoc ocr test.png --model mlx-community/Qwen2.5-VL-3B-Instruct-4bit
  ```
• Step 2: If Qwen works → problem is `EZCon/GLM-OCR-8bit-mlx` specific
  • Try `mlx-community` GLM-OCR model if one exists
  • Or re-quantize from the original model
• Step 3: If Qwen also fails → debug `OCRPipeline.processImage` (UserInput, prompt, generate params)
• Step 4: Verify safetensors header is valid (not zero-filled, see OCR: hf CLI xet download produces corrupted (zero-byte) safetensors #68)

Risks

• `EZCon/GLM-OCR-8bit-mlx` may simply be a bad quantization — out of our control
• GLM-OCR as a model may require specific prompt format not handled by MLXVLM's processor

[kiki830621]

Diagnosis (round 2 — refreshed after #84's backend refactor)

Type

bug (P1) — refreshed 2026-04-21 after significant landscape changes since original 2026-03-28 diagnosis.

What's changed since original diagnosis

The OCR subsystem has been substantially refactored since this issue was opened without explicitly re-verifying #66's symptom:

Date	Change	Effect on this issue
51e8a07	feat(config): add config ocr subcommand with host profile support	Added Ollama host profile management. Confirms Ollama path is the intended production fallback.
f1b975f	refactor: simplify PDF pipeline — replace block segmentation with page-level GLM-OCR	pdf ocr subcommand still defaults to EZCon/GLM-OCR-8bit-mlx. This path is still exposed to the #66 symptom.
1a95969 (#84)	wip: swap default OCR model to Qwen3-VL + OCRCore backend abstraction	Default model for top-level macdoc ocr shifted from EZCon/GLM-OCR-8bit-mlx to mlx-community/Qwen3-VL-4B-Instruct-4bit. OCRBackend protocol introduced; MLXBackend + OllamaBackend as implementations.
692dc17 (#79)	ocr-swift package extracted from macdoc	OCR code now lives in PsychQuant/ocr-swift; backend files under Sources/OCRCore/Backends/.

Current code state (MLXBackend.swift:8-11)

/// Note: As of mlx-swift-lm 2.31.3, VLM inference crashes with
/// broadcast_shapes error (ml-explore/mlx-swift-lm#191).
/// Use `OllamaBackend` as a workaround until upstream is fixed.

This docstring confirms a broader MLX VLM instability, not just GLM-OCR. If mlx-swift-lm#191 is the ongoing upstream crash, then any MLX VLM model (including Qwen3-VL) may crash or degrade.

The default wiring (MacDoc+OCR.swift:62) also reveals state:

let repo = model ?? (config.ocrDefaultModel == "glm-ocr"
    ? "mlx-community/Qwen3-VL-4B-Instruct-4bit"
    : config.ocrDefaultModel)

When user config says glm-ocr, macdoc silently routes to Qwen3-VL instead — a compatibility shim added to keep existing user config working while avoiding GLM-OCR. That alone is evidence the original bug was not fixed, just side-stepped at the default-model level.

Root Cause (refined)

Two independent causes, both still present:

1. GLM-OCR specific: EZCon/GLM-OCR-8bit-mlx quantization produces garbage output via MLXVLM. MLXVLM's GlmOcr.swift architecture exists but was never validated against this particular 8-bit quantization. Root cause candidates:

   • Bad quantization weights (most likely — it's a third-party quant, not mlx-community/*)
   • Tokenizer / chat-template mismatch (MLXVLM's Glm46VProcessor may expect a specific prompt format the model wasn't trained to handle)
   • Image preprocessing mismatch (unlikely — UserInput.Image.ciImage(...) is standard across MLXVLM models)

2. General MLX VLM instability (upstream mlx-swift-lm#191): broadcast_shapes crash affects VLM inference in general, not just GLM-OCR. This is an Apple / ml-explore issue that macdoc inherited. The MLXBackend docstring acknowledges it and recommends OllamaBackend as workaround.

Impact (updated)

• macdoc pdf ocr (Phase 2 pipeline) still defaults to GLM-OCR → still affected
• macdoc ocr (top-level, #84 era) defaults to Qwen3-VL → unclear whether Qwen3-VL via MLX currently works or silently broken due to #191
• macdoc ocr --backend ollama path → known-working fallback (per MLXBackend docstring), but requires user to run local Ollama server
• Files affected:
  • Sources/MacDocCLI/MacDoc+PDF.swift:203 (GLM-OCR default)
  • Sources/MacDocCLI/MacDoc+OCR.swift:60-65 (compatibility shim)
  • Sources/MacDocCLI/PageOCRRunner.swift:22 (Qwen3-VL default)
  • packages/ocr-swift/Sources/OCRCore/Backends/MLXBackend.swift (broadcast_shapes bug marker)

Strategy

Split into three decoupled tracks — each resolves independently:

Track A — Reproduce to confirm current state (investigation, no code)

• A.1 Install ollama locally (brew) + pull a VLM capable of OCR (e.g. ollama pull llava:latest or glm-ocr if available as Ollama model). Run macdoc ocr /tmp/ocr-test.png --backend ollama — establishes known-working baseline.
• A.2 Run macdoc ocr /tmp/ocr-test.png --backend mlx --model mlx-community/Qwen3-VL-4B-Instruct-4bit — does Qwen3-VL currently produce garbage, or does it work? This answer gates whether the MLX backend is entirely broken or only GLM-OCR is broken.
• A.3 Run macdoc ocr /tmp/ocr-test.png --backend mlx --model EZCon/GLM-OCR-8bit-mlx — confirm original OCR: GLM-OCR model produces garbage output via MLXVLM #66 symptom is still reproducible.
• A.4 Capture output samples + stderr logs to the issue as evidence for whatever fix path is chosen next.

Track B — GLM-OCR weight sanity (if A.3 confirms GLM still broken)

• B.1 Compare EZCon/GLM-OCR-8bit-mlx safetensors header + tokenizer config against any mlx-community/GLM-OCR-* variant (if one exists on HF by 2026-04-21).
• B.2 Try a different GLM-OCR quant or the non-quantized original via huggingface_hub download + re-quantize locally via mlx-vlm.
• B.3 If no working GLM-OCR quant exists anywhere → deprecate the EZCon/GLM-OCR-8bit-mlx default in MacDoc+PDF.swift:203; switch pdf ocr default to Qwen3-VL (matching macdoc ocr).

Track C — Upstream MLX-swift-lm bug tracking (if A.2 confirms general MLX path broken)

• C.1 Check ml-explore/mlx-swift-lm#191 status. If still open, document in macdoc CLAUDE.md + SECURITY.md the "Ollama preferred for OCR until upstream fix" recommendation.
• C.2 Consider promoting OllamaBackend to default when mlx-swift-lm hasn't moved past 2.31.3. Add a startup check that warns the user if MLX + an affected version are detected.
• C.3 Add a swift test case in ocr-swift that exercises MLXBackend.load(...) with a trivially small model (mocked or tiny VLM) to catch regressions — currently Tests/OCRCoreTests/ has only PDFKitExtractorTests.swift, no backend coverage.

Track D — Close-path decision

Depending on Track A findings:

• If Qwen3-VL works but GLM-OCR broken: fix is deprecate GLM-OCR default → update MacDoc+PDF.swift:203; close OCR: GLM-OCR model produces garbage output via MLXVLM #66 as "GLM-OCR quantization is the broken third-party artefact; macdoc now uses Qwen3-VL everywhere".
• If neither works via MLX: fix is promote Ollama; document in release-flow; close OCR: GLM-OCR model produces garbage output via MLXVLM #66 as "MLX-side OCR blocked by upstream mlx-swift-lm#191; Ollama is the production path; re-open once upstream moves".
• If both MLX paths work now: someone already silently fixed it upstream; close OCR: GLM-OCR model produces garbage output via MLXVLM #66 as resolved with reproduction evidence.

Complexity

SDD-warranted. Reasons:

• ✅ Cross-package: ocr-swift + macdoc CLI + possibly ml-explore/mlx-swift-lm (upstream only, not our code)
• ✅ Multiple decision points: (a) GLM-OCR fate (deprecate vs fix), (b) MLX-vs-Ollama default, (c) whether to add backend smoke tests to ocr-swift
• ✅ Affects 2 existing specs: simplified-pdf-ocr, mlx-model-management (both in openspec/specs/)
• ✅ Phased investigation (Track A must complete before Track B/C) — genuine ordering dependency

/spectra-discuss not strictly required IF Track A results make Track D's decision path obvious. But without Track A evidence, the "direction is unclear" triggers spectra-discuss as the safe default.

Risks

1. Reproduction requires model downloads (several GB) → Track A has non-trivial local setup cost. Mitigation: check if ~/.cache/huggingface/hub/ already has the models from prior sessions; reuse if possible.

2. mlx-swift-lm#191 may be a moving target → upstream version check needed at A.2 time. Pin the mlx-swift-lm version reported in Package.resolved to reproduce rather than implicitly using latest.

3. Ollama requires a running server → Track A.1 may fail on a fresh machine. Acceptable: document the setup in Track C.2's startup warning.

4. GLM-OCR deprecation is user-visible → users relying on EZCon/GLM-OCR-8bit-mlx via config ocr set-model glm-ocr will silently get Qwen3-VL (already happening per MacDoc+OCR.swift:62). Not technically a new break, but worth documenting in the close comment.

5. Tests in ocr-swift would need model fixtures → can't download multi-GB models in CI. Mitigation: Track C.3 uses mocks or a trivially small tokenizer-only test.

6. Issue is already 3 weeks stale → original evidence may have shifted (HF quantizations get re-uploaded, upstream versions bump). Track A is the epistemic gate.

Open Questions (for /spectra-discuss)

1. Track A order — run all 3 A.x reproductions today, or just A.2 (Qwen3-VL status) first?
2. GLM-OCR deprecation — if Track A.3 confirms GLM-OCR still broken, do we (a) keep trying to fix it, (b) deprecate entirely, or (c) document as "use Ollama for GLM-OCR" workaround?
3. MLX vs Ollama default — if both MLX models are flaky, should the default --backend switch to ollama? Risk: ollama setup burden on users who don't have a server.
4. Test coverage — ocr-swift/Tests/OCRCoreTests/ has only PDFKit; should we add a backend mock test as part of this fix to prevent silent regressions?
5. macdoc pdf ocr vs macdoc ocr — should they share the same backend default, or keep pdf ocr on GLM-OCR for some reason (e.g. higher accuracy on dense-text PDFs)?
6. Issue scope — is OCR: GLM-OCR model produces garbage output via MLXVLM #66 about "GLM-OCR model produces garbage" (narrow, the original title) or "MLX VLM OCR path unreliable" (broad, current state)? Decision affects close criteria.

Next

Complexity = SDD-warranted, 6 open questions, investigation gate before implementation.

Recommend: /spectra-discuss with topic "OCR backend strategy post-#84: resolve GLM-OCR garbage output + upstream MLX VLM instability — Track A investigation + default-model decision".