Skip to content

feat: support llmman as an Ollama-compatible local backend #127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ericcurtin wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: support llmman as an Ollama-compatible local backend #127

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,11 @@ NativeMind supports **Chrome**, **Firefox**, **Brave**, and **Edge** browsers wi
- **WebAssembly-Powered**: Runs directly in browser using WebAssembly
- **Trial Experience**: Perfect for evaluating the extension before setting up Ollama

### 📦 **[llmman](https://github.com/llmmanorg/llmman) (Ollama-compatible)**
- **Same API, Different Port**: llmman is a local model runner that serves the Ollama API (alongside OpenAI- and Anthropic-compatible ones) on port 17434
- **OCI & Hugging Face Models**: Pull models as OCI artifacts from any registry or directly from Hugging Face (`hf.co/org/model`), served by `llama.cpp`, `vllm` or `mlx-lm`
- **Setup**: Run `llmman serve` and `llmman pull gemma4`, then in NativeMind's Ollama settings set the **Server Address** to `http://localhost:17434/api`

## 🔮 Why On-Device AI？

### ⚠️ **The Problem with Cloud AI**
Expand Down Expand Up @@ -259,6 +264,7 @@ The built extension will be in the `.output/` directory.
**AI Models Not Working**
- **WebLLM**: Ensure sufficient RAM (8GB+ recommended) and modern browser
- **Ollama**: Verify Ollama is running locally on port 11434
- **llmman**: Verify `llmman serve` is running on port 17434 and the Ollama Server Address is set to `http://localhost:17434/api`
- Clear browser cache and restart extension
- Check browser console for error messages

Expand Down
2 changes: 1 addition & 1 deletion public/rules.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"requestHeaders": [{ "header": "Origin", "operation": "remove" }]
},
"condition": {
"regexFilter": "^https?://(?:localhost|127\\.0\\.0\\.1):11434/.*",
"regexFilter": "^https?://(?:localhost|127\\.0\\.0\\.1):(?:11434|17434)/.*",
"resourceTypes": ["xmlhttprequest"]
}
}
Expand Down
3 changes: 2 additions & 1 deletion utils/web-request.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ const RULE_ID_REMOVE_DISPOSITION = 2
export function registerDeclarativeNetRequestRule() {
// firefox has some bugs with declarativeNetRequest API, we use rules.json instead
if (import.meta.env.FIREFOX) return
const OLLAMA_URL_FILTER = /https?:\/\/[^/]*:11434\/.*/
// 11434: Ollama, 17434: llmman (serves the Ollama API, https://github.com/llmmanorg/llmman)
const OLLAMA_URL_FILTER = /https?:\/\/[^/]*:(?:11434|17434)\/.*/
const { resolve, promise } = Promise.withResolvers<void>()

const timeout = setTimeout(() => {
Expand Down