Spike: Apple Foundation Models + MLX local-provider lab for Pi-Rogue

[bearmug]

Summary

Add an experimental Local AI Platform Lab for Pi-Rogue to benchmark and optionally route cheap local reasoning tasks through Apple/MLX backends before escalating to the current Qwen 35B MoE expert path.

This is not about pretending we can get qwen3.6-35b-a3b to single-stream decode at 600 tok/s on one Mac. The real goal is to improve effective wall-clock speed and cost by using cheaper local models for routing, compression, JSON policy, context triage, and advisor preflight/review signals.

Motivation

Pi-Rogue is increasingly becoming an agentic orchestration system, not just a command bundle:

• /advisor needs better cheap mid-task signal.
• /goal, /loop, /autoresearch, and /autoresearch-lab need better local control loops.
• The context broker should keep prompts compact and avoid re-inserting huge payloads.
• The Skill Lifecycle / Forge work will need cheap intent resolution and workflow matching.
• The advisor binary gate retraining work needs better trajectory-health features and cheaper local escalation decisions.

Current baseline from #101 includes:

local/qwen3.6-35b-a3b-q4-mtp
Q4 + MTP
128K context
~3B active params
llama.cpp / llama-swap serving

This ticket adds a controlled lab path to compare that baseline against Apple Silicon-native options.

Hypothesis

The winning architecture is probably not:

everything -> Qwen 35B

It is more likely:

cheap local brainstem:
  route / compress / validate JSON / decide confidence / detect stuck trajectory

Qwen 35B MoE:
  hard code reasoning / repo synthesis / deep advisor reviews

frontier API:
  rare fallback / SOTA review / calibration

So the benchmark should measure end-to-end task wall time and Qwen-call reduction, not only raw tok/s.

Proposed scope

1. Add optional local-provider adapters

Add experimental provider shims behind an opt-in flag:

PI_LOCAL_AI_LAB=1

Candidate providers:

type LocalAiLabProvider =
  | "current-qwen"
  | "mlx-qwen"
  | "apple-fm";

Suggested behavior:

• current-qwen: existing local Qwen path, used as baseline.
• mlx-qwen: OpenAI-compatible or HTTP wrapper around mlx_lm.server.
• apple-fm: optional Mac-only wrapper around Apple Foundation Models / fm CLI / Python SDK.

All adapters should be optional. Non-macOS or unsupported macOS should skip apple-fm with an explicit reason, not fail the whole benchmark.

2. Add a small benchmark script

Suggested script:

npm run local-ai:bench

or:

npx tsx scripts/bench-local-ai-platform.ts

Suggested env/config:

PI_LOCAL_AI_LAB=1
PI_LOCAL_AI_PROVIDER=current-qwen|mlx-qwen|apple-fm
PI_MLX_BASE_URL=http://localhost:8080/v1
PI_APPLE_FM_ENABLED=1
PI_LOCAL_AI_RESULTS=.pi-rogue/local-ai-lab/results.jsonl

Output JSONL rows should include:

interface LocalAiBenchResult {
  id: string;
  ts: string;
  provider: "current-qwen" | "mlx-qwen" | "apple-fm";
  taskKind:
    | "intent_route"
    | "json_policy"
    | "context_compress"
    | "advisor_preflight"
    | "advisor_review"
    | "repo_triage"
    | "code_reasoning";
  promptTokens?: number;
  outputTokens?: number;
  wallMs: number;
  ttftMs?: number;
  decodeTokPerSec?: number;
  jsonValid?: boolean;
  passed: boolean;
  escalatedToQwen?: boolean;
  reason?: string;
}

3. Benchmark matrix

Use a compact task set that reflects Pi-Rogue reality:

Cheap/local tasks

These should often be answerable by Apple FM or a small local model:

• classify user intent into advisor/orchestration/forge/repo-memory/safety
• produce strict JSON policy object
• summarize compact context broker artifact
• decide whether a trajectory is stuck
• classify advisor review concern as high/medium/low/noise
• validate whether a proposed command is read-only/local/risky

Hard tasks

These should usually escalate to Qwen 35B:

• multi-file code reasoning
• repo synthesis
• debugging failed tests
• non-trivial architecture review
• deep PR review response
• ambiguous safety-sensitive shell action

4. Add routing experiment

Prototype a simple confidence policy:

interface LocalAiRoutingDecision {
  action: "answer_directly" | "compress_then_continue" | "escalate_to_qwen" | "escalate_to_frontier";
  confidence: number;
  reason: string;
  safety: "safe" | "needs_approval" | "blocked";
}

Initial policy:

confidence >= 0.85 and safe cheap task:
  answer locally

confidence 0.55 - 0.85:
  compress / summarize, then escalate to Qwen

hard code/repo/safety task:
  escalate to Qwen or existing advisor path

invalid JSON / malformed policy:
  fallback to current provider

No provider should silently run shell commands or change external state.

5. Keep this as a lab, not product default

Non-goals:

• Do not change default /advisor, /goal, /loop, /autoresearch, or /autoresearch-lab behavior.
• Do not make Apple FM or MLX required dependencies.
• Do not claim or target 600 tok/s single-stream Qwen decode.
• Do not convert Qwen to Core AI in this ticket.
• Do not add hidden auto-escalation to frontier models.
• Do not store raw prompts/responses unless explicitly configured.

Suggested implementation slices

Slice A — benchmark harness only

• Add scripts/bench-local-ai-platform.ts.
• Add static prompt fixtures.
• Run current-qwen baseline if configured.
• Emit JSONL + markdown summary.

Slice B — MLX adapter

• Document local setup:

pip install mlx-lm
mlx_lm.server --model mlx-community/Qwen3.6-35B-A3B-4bit-DWQ --port 8080

• Add HTTP client adapter.
• Run same prompt fixtures against MLX.

Slice C — Apple FM adapter

• Add optional adapter that shells out to fm CLI or Python helper.
• Skip cleanly when unavailable.
• Restrict to cheap local tasks only.
• Enforce strict JSON validation and fallback.

Slice D — routing policy simulation

• Run cheap-local-first routing against the fixture set.

• Report:

  • Qwen calls avoided
  • invalid JSON rate
  • wrong direct-answer rate
  • average wall time
  • p50/p95 wall time
  • escalation reasons

Acceptance criteria

• New lab benchmark can run without changing default Pi-Rogue behavior.
• Benchmark can compare at least current-qwen and one optional provider.
• Apple FM path is optional and skipped cleanly when unavailable.
• Results are written as JSONL with provider/task/timing/pass/escalation metadata.
• Markdown summary is generated with p50/p95 wall time and pass rate by provider.
• At least 20 fixture prompts cover routing, JSON policy, context compression, advisor review, and hard-code escalation.
• Invalid JSON from a cheap provider falls back to current provider.
• Hard repo/code tasks do not get answered directly by cheap provider unless explicitly allowed.
• No shell command or external state-changing action is executed by this lab.
• Docs include honest limitation: goal is effective throughput/cost reduction, not 600 tok/s single-stream Qwen decode.

Related work

• Retrain advisor binary gate for agentic trajectories (fix preflight-always-escalate / review-always-continue collapse) #101 advisor binary gate retraining
• PRD: Skill Lifecycle Engine + Forge Recipes for Pi-Rogue #102 Skill Lifecycle Engine + Forge Recipes
• Follow up: complete context broker roadmap after durable prompt-load beta #89 context broker roadmap

Open questions

1. Should this live under scripts/ only, or become a small internal packages/core provider abstraction?
2. Should MLX be treated as an OpenAI-compatible endpoint only, or should we add a direct Python bridge later?
3. Should Apple FM be used only for routing/compression, or also for low-risk direct advisor answers?
4. What is the minimum benchmark set that predicts real Pi-Rogue usefulness without burning a full overnight run?
5. Should the results feed into advisor binary gate retraining as additional trajectory features?