InferX

Private On‑device LLM · Cross‑platform Inference SDK · Zero Cloud

Powered by llama.cpp · Lightweight • Fast • Extensible

中文文档

---

📚 Table of Contents

• Why InferX
• Updates
• Features
• Architecture
• Platforms
• Install & Build
• Quickstart
• Examples
• Benchmarks
• Roadmap
• FAQ
• Contributing
• License

---

✨ Why InferX

• 🔒 Privacy‑first: fully offline, data never leaves the device.
• ⚡️ Low‑latency streaming: token‑by‑token output with smart UTF‑8 assembly.
• 🧩 Stackable LoRA: load multiple adapters, tune scales dynamically.
• 🔌 OpenAI‑compatible: Chat Completions with tools/tool_choice and sampling controls.
• 📦 Unified C interface: one llx.h for desktop and mobile, wrapped for native and Flutter.
• 🧱 Production‑ready samples: Android AAR, iOS Swift Package, CLI/Agent, Flutter plugin and apps.

---

🆕 Updates

• 2025‑10: Initial public release and multi‑platform samples.

---

💡 Features

• Streaming/non‑streaming generation with KV reuse/clear and UTF‑8 safe chunks.
• OpenAI‑compatible Chat API: llx_chat_complete_json supports messages/tools/tool_choice/temperature/top_p/top_k/max_tokens.
• Multi‑LoRA stacking: llx_session_add_lora, _update_lora_scale, _remove_lora, _clear_lora.
• Unified C API: backend init, model/session lifecycle, stepwise generation, benchmarking.
• Comprehensive samples: CLI chat and local Agent, Android AAR + app, iOS SPM + app, Flutter plugin + app.

---

🧱 Architecture

Layers from top to bottom:

• Core: llama.cpp‑based inference with chat templates and tool‑calling parsing.
• C interface: unified llx.h across OS/languages.
• Bindings: Android (JNI/Kotlin), iOS (ObjC++/Swift), Flutter (MethodChannel).
• Tooling: local benchmarks (llx_bench), LoRA management, sample apps.

---

🧭 Platforms

OS/Framework	Native	Flutter	Notes
Android 8+	✅	✅	CPU backend; ABI: arm64‑v8a; Vulkan/NNAPI planned
iOS 15+	✅	✅	Integrate via Swift Package; Metal depends on binaries; CoreML planned
Windows 10+	✅	⏳	CPU path; DirectML planned
macOS 12+	✅	⏳	Metal/Accelerate (via backend)
Linux (x86/arm)	✅	⏳	CPU path; BLAS optional per llama.cpp docs

---

⚙️ Install & Build

Prerequisites

• CMake 3.20+, C/C++17 toolchains (Xcode/CLT on macOS; VS 2022 x64 on Windows).
• Optional: BLAS on Linux as per llama.cpp docs.

Desktop (macOS/Linux)

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Desktop (Windows, VS 2022)

cmake -S . -B build -G "Visual Studio 17 2022" -A x64
cmake --build build --config Release -m

Binaries will be in build/ (e.g., llx-cli, llx-agent).

Android (AAR and sample app)

1. Build the native AAR:

cd android/llx-android
./gradlew assembleRelease

Artifact: android/llx-android/build/outputs/aar/llx-android-release.aar

2. Run the sample app:

• Open android/ in Android Studio and run llx-example/app; or via CLI:

cd android
./gradlew :llx-example:app:assembleDebug

Current ABI is arm64-v8a; use an arm64 emulator or device.

iOS (Swift Package and sample)

• Add ios/llx-ios as a local Swift Package in Xcode (Add Local → select Package.swift).
• Place prebuilt native binaries (e.g., llama.xcframework) into ios/llx-ios/Sources/InferxLLMNative/ and ensure linkage in Package.swift (sample binaryTarget provided).
• See ios/Example/InferxLLMExample/ for the sample app.

Flutter (plugin and example)

• Plugin: flutter/llx_flutter (Android supported, iOS WIP).
• Add as a path dependency in your Flutter app:

dependencies:
  llx_flutter:
    path: ../../flutter/llx_flutter

• Run the example:

cd flutter/llx_flutter/example
flutter pub get
flutter run

Build llx-android first on Android.

---

🚀 Quickstart

CLI (llx-cli)

• Interactive chat:

./build/llx-cli --model /path/to/model.gguf --ctx 8192 --nlen 1024

• One‑shot JSON (OpenAI compatible):

./build/llx-cli --model /path/to/model.gguf --json ./request.json

• Multiple LoRA with per‑adapter scale:

./build/llx-cli --model /path/base.gguf \
  --lora /path/adapter1.gguf:0.2 \
  --lora /path/adapter2.gguf:0.4

Local Agent (function calling demo)

./build/llx-agent --model /path/to/model.gguf --ctx 16384 --max_tokens 1024 --temp 0.7 --top_p 0.9 --top_k 40

Android (Kotlin)

LLX.nativeInitBackend()
val model = LLX.nativeModelLoad(modelPath)
val sess = LLX.nativeSessionCreate(model, 8192, 0)
LLX.nativeSessionInitFromText(sess, "你好", true, 1024)
while (true) {
    val r = LLX.nativeSessionStep(sess, 1024)
    print(r.text)
    if (r.finished) break
}
LLX.nativeSessionFree(sess)
LLX.nativeModelFree(model)
LLX.nativeFreeBackend()

iOS (Swift)

import InferxLLMKit

let model = try InferxModel(path: "/path/to/model.gguf")
let sess = try model.createSession()
try sess.initFromText("Hello", formatChat: true, maxLen: 512)
while true {
    let (chunk, finished) = try sess.step(maxLen: 512)
    print(chunk, terminator: "")
    if finished { break }
}

Flutter (Dart)

final llx = LlxFlutter();
await llx.initBackend();
final model = await llx.modelLoad('/path/to/model.gguf');
final session = await llx.sessionCreate(model, nCtx: 8192, nThreads: 0);
final messagesJson = '[{"role":"user","content":"你好"}]';
await llx.sessionInitFromMessagesJson(session, messagesJson);
while (true) {
  final r = await llx.sessionStep(session);
  print(r.text);
  if (r.finished) break;
}

---

📦 Examples

• example/cli.cpp: CLI chat with JSON one‑shot and multi‑LoRA stacking.
• example/agent.cpp: Local function‑calling Agent, prints request/response JSON.
• android/llx-android: AAR + JNI binding.
• android/llx-example: Minimal native sample app with streaming UI.
• ios/llx-ios: Swift Package (CLLX/Native/Kit layers).
• ios/Example/InferxLLMExample: SwiftUI sample app.
• flutter/llx_flutter: Flutter plugin and example app (iOS in progress).

---

📈 Benchmarks

llx_bench helps measure prefill (pp) and generate (tg) throughput. Unified scripts and comparison tables will be added.

---

🗺️ Roadmap

• Unified Session API (streaming/non‑streaming)
• Android AAR + native sample
• iOS Swift Package + native sample
• Flutter plugin (Android) + sample
• Flutter (iOS)
• Vulkan/NNAPI/CoreML backends
• HarmonyOS support

---

❓ FAQ

Q: Multi‑LoRA sometimes degrades quality? Layer conflicts or improper scales may cause drift. Try lower scales, scenario‑specific adapters, or offline distillation/merging.

---

🤝 Contributing

PRs and issues are welcome. Please read this README and submodule READMEs (Android/iOS/Flutter), run the samples on your target platform, then file improvements or bug reports.

---

📄 License

Apache‑2.0

⭐️ If you find InferX useful, please give it a star! ⭐️