genie-ai-model

LoRA fine-tunes of small open-weights LLMs specialized for the GeniePod smart-home agent loop. Goal: bake system instructions, tool schemas, and dispatch patterns into the model so the runtime context can shrink from "general assistant + full toolspec + history" down to "user utterance + last 2–3 turns + memory summary."

Pairs with genie-ai-runtime (the Jetson Orin-tuned inference engine) and genie-claw (the home agent itself).

Target hardware: Jetson Orin Nano Super 8 GB, served via genie-ai-runtime. Base model (initial): Qwen3-4B Q4_K_M. Status: planning. No weights, training scripts, or evals checked in yet.

Why

genie-ai-runtime v0.1.0-alpha.12 reaches ~9.9 tok/s decode and ~877 ms cold TTFT for a 33-token prompt on a Jetson Orin Nano Super 8 GB. TTFT scales roughly linearly with prompt length — at 235 tokens it's ~6300 ms.

A naïve GeniePod request to the model carries:

• General assistant system prompt
• Full tool schema (light, door, thermostat, camera, coffee, scenes, …)
• N turns of conversation history
• A retrieved memory / RAG passage

That assembled prompt routinely lands at 1500–3000 tokens — i.e. ~40–80 s of cold TTFT on Orin Nano. The user perceives that as "the assistant is slow," even though decode is fast once it starts.

There are two ways out:

1. Engine-side: persistent KV (genie-ai-runtime Path F), draft models, chunked prefill, etc. Real, but bounded by how much you can keep cached and how much state varies per turn.
2. Model-side (this repo): teach the model the patterns so the runtime never has to send them. The system prompt, tool grammar, reasoning template — all internalized in the weights.

(2) is the lever that compounds with everything (1) does, because it shrinks N at the root.

Approach

LoRA / PEFT fine-tune on top of Qwen3-4B-Instruct (and later candidates like Phi-4-mini, Gemma-3n-E2B) using a domain-specific dataset of GeniePod-shaped traffic:

• Smart-home control utterances → tool-call outputs
• Multi-turn home-context conversations (memory + last-N-turns shape)
• Skill orchestration / multi-tool plans
• Refusals + clarifications in the GeniePod voice

At runtime the request shape collapses to:

[short system marker]
[memory summary, ~50–100 tokens]
[last 2–3 turns]
[user utterance]

…with the tool schema, dispatch grammar, and assistant persona all already in-weights.

Target gains

Measured against the genie-ai-runtime baseline above, on Qwen3-4B Q4_K_M:

Metric	Today (general prompt)	Target (after FT)	Why
Runtime prompt tokens	1500–3000	200–400	Schema + system + persona moved into weights
Cold TTFT	40–80 s	5–11 s	Linear in prompt length
Decode tok/s	9.9	9.9 (unchanged)	Same architecture, same KV
Tool-call accuracy on home tasks	TBD (eval to come)	≥ base + N pts	Domain-tuned
KV memory @ typical prompt	100–250 MB	15–35 MB	INT8 KV × shorter ctx

The decode-speed number is not the headline — TTFT and context footprint are. End-to-end "user finishes speaking → assistant starts speaking" is dominated by TTFT on Orin Nano.

Roadmap

Phase	Goal	Output
0. Scope	Define the GeniePod intent surface (every tool, every skill, every refusal class)	data/schema/
1. Data	Collect / synthesize a training set (target: 5–10k examples)	data/train/, data/eval/
2. Base eval	Measure stock Qwen3-4B on the eval set (accuracy + TTFT + token count)	evals/baseline.md
3. LoRA train	r=16 LoRA on instruct base, single A100 / RTX 4090 / Colab Pro	training/qwen3-4b-genie/
4. Eval	Same set vs. base — must improve tool-call F1, must not regress general-chat coherence	evals/lora-v1.md
5. Quantize	Re-quantize to Q4_K_M for Jetson	models/genie-qwen3-4b-Q4_K_M.gguf
6. Integrate	Wire into genie-ai-runtime, A/B vs. stock Qwen3-4B with shortened prompt	integration.md
7. Iterate	Move whichever benefit is biggest (latency / accuracy / footprint)	tagged releases

Non-goals

• Pretraining from scratch.
• Building a new inference engine (genie-ai-runtime does that).
• Multi-modal (vision/audio) — out of scope for v1.
• A general-purpose assistant — this model is intentionally narrowed.
• Public model hosting / API. The fine-tune ships as a GGUF for local use.

Related

• genie-ai-runtime — the inference engine that will serve this model on Jetson Orin.
• genie-claw — the home assistant that will issue the shortened prompts.

License

Code in this repo: Apache-2.0. Patent-grant clause matters for ML/research code, hence Apache over MIT (sibling genie-ai-runtime is MIT — different rationale, infrastructure rather than research).

Released model weights: subject to the upstream base model's license. Qwen, Phi, Gemma each carry their own terms; any GGUF we publish ships with a copy of the relevant upstream license alongside this Apache-2.0 license covering the training and packaging code. See NOTICE.