CoreAIKit

Build LLM and computer-vision apps on Apple's Core AI framework (macOS / iOS 27 beta) in a few lines of Swift.

Community package — not affiliated with Apple. Requires macOS 27 beta / iOS 27 beta (real device; the CoreAI framework is not in the iOS Simulator SDK).

import CoreAIKit

let chat = try await ChatSession(model: .qwen3_0_6B)
for try await event in chat.streamResponse(to: "Hello!") {
    if case .response(let delta) = event { print(delta, terminator: "") }
}

Models download automatically from the Hugging Face Hub on first use — no Python required.

See it running

Real-device captures (iPhone 17 Pro / M4 Max), everything fully on-device. Each cell links to the kit example — or zoo app — that runs the same model. (Media lives in coreai-assets, so cloning this repo stays fast.)

Chat — Youtu-LLM-2B ChatDemo	Speech→text — Whisper v3 turbo Transcribe	Who-said-what — Sortformer + Parakeet Meeting
Screen VLM — Holo2-4B VLChat	Repo agent — FastContext-4B zoo CoreAIChat	Detection — RF-DETR nano, no NMS DetectCamera
Segmentation — SAM 3 zoo	Depth — Depth Anything 3 DepthCamera	4× super-res — AdcSR UpscaleDemo
PII redaction — GLiNER2 InfoExtract	Doc OCR — GLM-OCR 0.9B, ~4 s/page ReadDoc	1.58-bit ternary — BitCPM-8B in ~2.1 GB zoo CoreAIChat

Text→image — GLM-Image zoo CoreAIImageGen	In-context edit — FLUX.2 klein zoo CoreAIImageGen
Text→video — LTX-Video 2B zoo CoreAIVideo	Photo→3D splat — TripoSplat zoo TripoSplatMac
Diffusion LLM (parallel denoise) — LLaDA-8B DiffuseChat	Doc→Markdown — MinerU2.5 ReadDoc

Time-series forecasting — TimesFM 2.5, ~25 ms/forecast on iPhone · Forecast

Agentic coding — Ornith-1.0-9B on M4 Max · zoo CoreAIChatMac

Works with Apple's FoundationModels API

KitLanguageModel plugs any catalog chat model into the system LanguageModelSession — the same FoundationModels API you use for Apple's built-in model — and adds what the stock CoreAILanguageModel adapter lacks: tool calling (ChatML/Hermes models) and guided generation (sequential engines).

import FoundationModels
import CoreAIKit

let model = try await KitLanguageModel(model: .qwen3_0_6B)   // downloads once, then cached
let session = LanguageModelSession(model: model, tools: [WeatherTool()])
let answer = try await session.respond(to: "What's the weather in Tokyo?")

Your Tool implementations, @Generable types, streaming snapshots, and transcripts work unchanged. See Examples/FMToolDemo and Examples/GuidedDemo.

What's inside

Product	What it gives you
CoreAIKit	ModelStore (download/cache), ModelCatalog (live model list), ChatSession (streaming chat + live stats + guided generation), KitLanguageModel (FoundationModels provider with tool calling + guided generation)
CoreAIKitVision	GraphModel (run any .aimodel), ImageTextEncoder (CLIP), DepthEstimator, CameraFeed, image preprocessing
CoreAIKitEmbeddings	TextEmbedder (EmbeddingGemma, 768-d normalized) for on-device search and RAG
CoreAIKitUI	SwiftUI components: ModelPickerBar, ChatTranscriptView, StatsBar

Examples

Text & chat

• Examples/ChatDemo — multiplatform chat app (~150 lines)
• Examples/DiffuseChat — diffusion LLM chat: watch LLaDA-8B denoise all tokens in parallel (Mac)
• Examples/FMToolDemo — local tool calling behind LanguageModelSession (swift run)
• Examples/GuidedDemo — guided generation: schema-valid JSON by construction (swift run)
• Examples/InfoExtract — schema-driven extraction / PII redaction with GLiNER2 (iPhone + Mac)

Vision

• Examples/VLChat — local VLM image chat (Qwen3-VL) via the KitVisionModel vision executor (iPhone + Mac)
• Examples/AskVLM — your Qwen3-VL as its own Visual Intelligence tab (offline "ask")
• Examples/VisualIntel — your own CLIP / RF-DETR behind the system Visual Intelligence search (iOS camera / iPad+Mac screenshot)
• Examples/PhotoSearch — semantic photo search with CLIP (iOS)
• Examples/DetectCamera — real-time object detection with RF-DETR, no NMS (iOS; nano 33–39 FPS end-to-end on iPhone 17 Pro via the zero-copy capture pipeline)
• Examples/DepthCamera — live camera depth with Depth Anything 3 (iOS)
• Examples/UpscaleDemo — one-step diffusion super-resolution with AdcSR
• Examples/ActionCamera — video action recognition with V-JEPA 2 (world model)
• Examples/ReadDoc — whole-page document OCR → Markdown (GLM-OCR / MinerU2.5)
• Examples/DocSearch — visual document retrieval, no OCR (ColModernVBERT late interaction)

Audio & speech

• Examples/Transcribe — speech→text (Whisper large-v3-turbo, Qwen3-ASR, Parakeet TDT)
• Examples/Meeting — who-said-what: Sortformer diarization + per-turn ASR in one API
• Examples/Speak — text-to-speech (Kokoro, VoxCPM)
• Examples/Music — text→music with Stable Audio Open Small (~12× realtime on iPhone)
• Examples/AudioChat — audio understanding — describe sounds, not just transcripts (Qwen2.5-Omni)

RAG, agents & system integration

• Examples/DocChat — on-device RAG over your notes: embeddings + retrieval tool + local LLM (swift run)
• Examples/SpotlightChat — local RAG with Apple's SpotlightSearchTool (WWDC26) behind your own model (swift run)
• Examples/SpotlightApp — the "ask your notes" RAG chat as a real SwiftUI app (iPhone + Mac), behind your own model
• Examples/SiriAsk — ask your local model from Siri (App Intents + onscreen awareness + risk-based confirmation; ≥4B)

Other modalities

• Examples/Forecast — time-series forecasting with TimesFM 2.5 (~25 ms/forecast on iPhone)

See docs/GETTING_STARTED.md.

Requirements

• macOS 27 beta / iOS 27 beta, Xcode 27 beta
• Models run fully on device

Versioning & stability

Tagged releases, SemVer, a pinned model catalog (each entry carries the verified Hugging Face revision), and CI + a nightly end-to-end gate on macOS 27 beta. See docs/STABILITY.md and CHANGELOG.md.

License

BSD-3-Clause. See LICENSE and NOTICE.txt (portions adapted from apple/coreai-models and john-rocky/coreai-model-zoo).