A desk robot that runs on a Raspberry Pi 5. It has a camera, an animated face on a touchscreen, a Bluetooth speaker, a USB mic array, and a small servo arm. You tap the screen and talk to it.
The interesting part isn't the hardware. It's how the intelligence is layered so the robot stays useful when the network is slow, rate-limited, or gone.
Cloud models are the smartest link and the least dependable one. Rate limits, network drops, latency. So the design puts deterministic reflexes and local models underneath the cloud instead of in front of it.
| Layer | What runs | Speed | Cost | Used for |
|---|---|---|---|---|
| Reflex | Scripted logic, no model | Instant | Free | Keyword to gesture, "is someone there", arm choreography |
| Cognition | Cloud models | 2 to 5 s | Quota-limited | Conversation, rich scene description, mood |
| Resilience | Models on the Pi | Slow or instant | Free, offline | Vision with no network |
Each layer can answer alone. Lose the cloud and the robot gets less clever, not dead.
Gemini is the eyes. Ask the robot what it sees and a frame goes to Gemini, which answers in a sentence or two: who's in the room, what's on the desk, what mood someone looks to be in.
Two properties make it the right pick for that job:
- It's multimodal. One call takes the image and the question together, so "what is that person holding" works without a separate detector feeding a text model
- It's fast enough to sit inside a spoken turn. Two to five seconds is a pause in conversation, not a dead air
Vision is where a general model earns its keep. A fixed object detector gives you labels. Gemini gives you a description someone would actually say out loud, which is what a robot with a voice needs.
Worth knowing: on the free tier you get a small daily request budget. Hit it and vision stops. The design treats that as normal and falls through to the next layer rather than failing the conversation.
Gemma runs on the Pi itself through Ollama. No network, no key, no quota.
It's the answer to a specific question: what should the robot say when someone asks what it sees and the internet is down? Gemma describes the frame locally. On a CPU-only Pi 5 that takes a couple of minutes, so it never runs inside a live call. It's the offline path, and it's worth the wait when the alternative is nothing.
The pairing is the point. Gemma is the same family of research as Gemini, small enough to run on hardware you can hold. You get a consistent quality of description across both paths, which matters when the fallback is meant to feel like the same robot rather than a degraded one.
ask what it sees
|
v
Gemini --------- answers -------> rich description
|
fails or out of quota
|
v
network up? -- yes --> on-device detector (instant labels)
|
no
v
Gemma on the Pi (slow, rich)
A quota error and a dead network are different problems and they get different answers. That distinction is most of what makes the thing feel reliable.
Everything above describes a robot on a desk. If you want to build on Gemini past a hobby footprint, Google Cloud is where that goes, and it changes several constraints at once.
Vertex AI is the managed home for Gemini. Moving from a studio API key to Vertex gets you:
- Real quota instead of a free-tier daily cap, billed per token, with quota you can request more of
- Regional endpoints, so you pick where inference happens and keep data in a region for compliance
- IAM and service accounts rather than a long-lived key on a device. The robot holds a scoped credential, not a master key
- Provisioned throughput if you need predictable latency under load
- Audit logs, which you need the moment more than one person uses the thing
The rest of the stack has obvious homes too:
- Cloud Run for the agent API. It's HTTP, it's stateless, it scales to zero. A device-facing service that idles most of the day is close to free
- Secret Manager for keys, so nothing sensitive lives in a file on the robot
- Model Garden if you want to serve Gemma yourself, on a GPU, instead of running it on the device. Same model, different latency budget
- Cloud Storage and Firestore for anything you want to keep across reboots
- Lyria on Vertex for music generation, which needs a billed project
You'd want this the moment there's more than one robot, or more than one person, or a requirement you have to prove rather than assert. For a single device on a desk, an API key and a Pi are enough, and the layered fallback above matters more than any of it.
- Raspberry Pi 5, 8 GB, NVMe boot
- CSI camera for vision
- DSI touchscreen for the face
- USB far-field mic array for input
- Bluetooth speaker for output, on A2DP
- ESP32-based servo arm over USB serial
Splitting the mic from the speaker is worth calling out. When one Bluetooth device does both, it runs a headset profile: mono, narrowband, and everything the robot says sounds like a phone call. A separate USB mic frees the speaker for full-bandwidth audio. The cost is that you lose the headset's built-in echo cancellation, so the robot can hear itself through the speaker. Plan for that.
A few things that cost real debugging time.
Fail loudly at the edges. A tool call that returns a healthy 200 while doing nothing is worse than an error. Several bugs here looked like personality problems because the robot talked normally and silently did nothing.
Don't name devices in health checks. Asking "is the Bluetooth speaker present as a microphone" breaks the day you add a different microphone. Ask "is there a microphone" instead.
A servo with no feedback can't tell you where it is. Any angle you command is a guess about the physical world. Guess wrong and the arm points at a wall. Where that matters, the code refuses to move until a person confirms the bearing.
Deterministic beats clever for anything safety-adjacent. The stop command is a keyword match on the transcript, not a model decision.
src/camera_agent.py the service: camera, vision, tools, arm, audio, the loops
src/prep_agent.py matches calendar events to the mail that is about them
src/face.html the kiosk UI: animated face, voice client, video player
src/edge/ the Cloudflare Worker that fronts the tunnel
src/test_camera_agent.py the suite
systemd/ units, timers, and the two self-healing scripts
pipewire/ audio routing: mic selection, A2DP, echo cancellation
Running the suite:
pip install -r requirements-test.txt
cd src && pytest test_camera_agent.pyNo camera, no servo, and no Pi needed. OpenCV, pyserial and the model runtime
are stubbed by conftest.py when they are missing, and the handful of tests
that read the private assistant config skip instead of failing.
The tests are the useful part if you're reading for ideas. Most of them are named after a bug that actually happened, with a comment explaining what broke. A test named after the fault it prevents survives a refactor. One named after the function it covers does not.
Some things are deliberately absent: the assistant prompt, saved memory, arm calibration, and anything the camera or the mic produced. See Privacy below.
No captured images, no generated audio, no calendar or mail content, no secrets, and no assistant prompt. The personality, the memory, and everything the robot has seen or heard stay on the device.
The published code is sanitised: usernames, device addresses, and workspace ids are placeholders, and a scheduled personal reminder is generalised. Nothing was removed that changes how the system works.
Free for personal use and student research. Commercial use requires a paid license. Attribution to Gretchen Boria, PhD.
See LICENSE for terms.