The brain of Phil, an AI drummer robot.
Voice in → Whisper STT → LLM classifier / planner → validated robot commands → MeloTTS voice out.
phil-interaction turns spoken Korean into safe, executable robot behavior. It listens on a microphone, transcribes with Whisper, classifies the intent with a small LLM, plans a sequence of skills with a larger one, validates every step against the robot's state and motor limits, and streams the resulting commands to the C++ controller (phil-control) over TCP while talking back through MeloTTS.
- Two-stage LLM decision layer: a fast intent classifier followed by a domain planner. The backend is chosen per model name, either an OpenAI-compatible API or local Ollama (Qwen3 on Jetson AGX Orin), via
PHIL_CLASSIFIER_MODEL/PHIL_PLANNER_MODEL. - Turn state machine (
pipeline/robot_fsm.py,pipeline/brain_pipeline.py): prefilter → classify → direct answer or plan → execute → return home, with a home watcher that brings the robot back to a safe pose. - Skill expansion and validation (
skills.py,motion_resolver.py,command_validator.py,validator.py): plans are expanded into concrete motion commands and checked against motor configuration and current robot state before anything is sent. - Interruptible execution (
pipeline/exec_thread.py): commands run on a background thread; a new utterance preempts the current action. - Runtime layer (
runtime/): microphone listener, TCP client for the controller'sOPCODE|arg|argprotocol, MeloTTS engine, console logging. - Evaluation (
eval/): offline smoke and scenario suites, classifier and planner model benchmarks, latency isolation, voice I/O benchmark, with reports checked in.
mic ─▶ Whisper ─▶ prefilter ─▶ intent classifier ─▶ planner ─▶ skill expansion ─▶ validator ─▶ TCP ─▶ phil-control
│ ▲
└─▶ direct answer ─▶ MeloTTS ─▶ speaker GET_STATUS ─┘
phil_robot/
├── phil_brain.py # entry point: STT, state snapshot, turn FSM, executor, TTS
├── config.py # model names, backend detection, runtime knobs
├── pipeline/ # decision layer: classifier, planner, validator, skills, FSM, executor
├── runtime/ # mic listener, TCP robot client, MeloTTS engine, console log
├── prompts/ # classifier / planner prompt templates
├── data/ # skill, song and motion tables consumed by the pipeline
├── eval/ # offline evaluation runners, benchmark cases and reports
├── tests/ # unit tests
├── docs/ # architecture, contracts, benchmarks (KR / EN)
├── assets/ # runtime assets
├── third_party/ # vendored MeloTTS
└── environment.yml # conda env `drum4`
conda env create -f environment.yml
conda activate drum4
# Ollama serving the configured models (or set PHIL_*_MODEL to an OpenAI model)
export PHIL_CLASSIFIER_MODEL=qwen3:4b-instruct-2507-q4_K_M
export PHIL_PLANNER_MODEL=qwen3:30b-a3b-instruct-2507-q4_K_M
# start phil-control first, then:
python phil_brain.pyOffline evaluation without a robot:
python eval/run_eval.py --suite smoke- LLM pipeline architecture · KR
- Project structure · KR
- Sequence diagram (KR) · Thread lifecycle (KR) · State machine (KR)
- Contracts · Function specs (KR)
- Classifier benchmark report · KR · Planner benchmark round 1 (KR)
| Repo | Role |
|---|---|
| phil-control | C++17 real-time body controller: state machine, trajectories, TMotor / Maxon / Dynamixel |
| phil-interaction (this repo) | Python brain |
| phil-simulation | Frame-level PyBullet SIL fed by the controller's raw CAN frames and Dynamixel packets |
| phil-midi-converter | MIDI ↔ score converter: builds Phil's text scores from Groove MIDI Dataset drum tracks |
| phil-matlab-analysis | MATLAB log analysis: motor tracking plots, Simscape Multibody replay of logged motion, OBB/SAT collision check |
Developed at KIST.
phil-interaction은 Phil 드럼 로봇의 두뇌입니다. 마이크 입력을 Whisper로 받아 적고, 작은 LLM으로 의도를 분류한 뒤 큰 LLM으로 스킬 시퀀스를 계획합니다. 계획은 모터 설정과 현재 로봇 상태에 맞춰 검증된 후 TCP로 C++ 제어기에 전달되고, 답변은 MeloTTS로 재생됩니다. 세부 구조와 계약, 벤치마크는 docs/를 참고하세요.