Spec H — Multimodal Intent Contract & 2026-Q3 Roadmap

[broomva]

Umbrella

This issue is the umbrella for the 22 actionable sub-tickets across the 7 tracks defined in docs/specs/2026-05-13-mil-roadmap-competitive-landscape.md (sensorium repo, commit 49fbc49, PR #10).

The contract

MIL declares a Multimodal Intent Contract with three load-bearing primitives — already shipped:

1. Directive<S> typestate (pneuma-core) — Composing / Ready / Proposed / Committed lifecycle with the five categorical guarantees.
2. StreamUpdate<T> + Generation substrate (sensorium-core) — generation-tagged Partial/Final/Cancelled stream events so downstream stages can speculate + cancel cleanly.
3. PrimitiveToken family (sensorium-core) — Reference/Predication/Modulation/Relation/Approval/Attention/State, each produced by a different modality.

Spec H formalizes the contract for publication as the open canonical document, modelled after Spec E — Agent-Loop Compute Contract. Strategic positioning: same play as Spec E at silicon, one substrate layer up — own the contract, let backends compete underneath.

Seven tracks

Track	Scope	Window	Q3 priority
C	Streaming substrate completion (dialog crate, barge-in, provenance event log, streaming agent kickoff)	2026 Q3	Urgent — C1, C2
B	Voice backend optionality (Apple SpeechAnalyzer ✦, Kyutai, Whisper-turbo, Moonshine, Parakeet-v3, OpenAI Realtime, Gemini Live)	2026 Q3-Q4	Urgent — B-Apple
T	TTS substrate via Sesame CSM 1B (cartesia-one/csm.rs drop-in) + Kokoro edge + audio output	2026 Q3-Q4	Urgent — T1, T2
V	Vision substrate (ScreenCaptureKit + Moondream 3 + Florence-2 + VLM-as-backend + resolver bridge)	2026 Q4-2027 Q1	High — V1
N	Non-voice modalities (Mediapipe hands MVP → Tobii gaze → Vision Pro ARKit)	2026 Q4-2027 Q2	High — N1
P	Cryptographic provenance (Ed25519 signing, audit-trail UX, replayable bundles)	2026 Q4	—
G	Product surfaces (video demo, Pointer SDK, voice keyboard firmware, App Intents bridge)	2027	—

Q3 critical path

1. C1 — pneuma-dialog crate (extracts the inline VAD loop from pneuma-demo into a reusable orchestration driver)
2. C2 — Barge-in via VoiceSession::cancel() (substrate ready since B2)
3. B-Apple — Apple SpeechAnalyzer (macOS 26 Tahoe, 2× faster than Whisper-turbo, on-device default)
4. T1 + T2 — sensorium-speech crate + Sesame CSM 1B via cartesia-one/csm.rs
5. G1 — 2-minute live realtime video demo (highest-ROI marketing asset; the polycast5 reel got 7K likes in a day)

Six decisive insights (from the spec)

1. Voice/multimodal input is the converged 2026 competitive surface (AI Pointer, Realtime-2, Tahoe, Granola $1.5B, Wispr $81M).
2. Nobody ships a cross-vendor portable typed substrate. Apple App Intents is closest but vertically locked.
3. Cloud-blob vs composed-pipeline duopoly leaves on-device unattended. MIL + Parakeet + Llama + CSM + Moondream = the missing turnkey on-device stack.
4. Apple SpeechAnalyzer (macOS 26) is a forcing function — MIL must plumb to native, not compete.
5. Vertical integration is the only durable hardware play. Standalone wearables die (Humane / Friend / Bee→Amazon / Limitless→Meta).
6. Provenance is structurally absent from the entire market. Ed25519 signing is a Yubico-style structural moat.

Linked artifacts

• Spec: docs/specs/2026-05-13-mil-roadmap-competitive-landscape.md
• Commit: 49fbc49
• PR: sensorium#10 (merged)

Sub-tickets follow below this umbrella. Each carries acceptance criteria, priority, and dependency links from the spec.

[linear]

BRO-1064