From 9aca7f3a16d622997975a1c956b0929b77c12532 Mon Sep 17 00:00:00 2001 From: Massimiliano Fiori Date: Sun, 6 Sep 2026 11:59:13 +0200 Subject: [PATCH 1/2] docs: two Qwen 2.5 Coder 14B transcripts of find-and-kick, the local model evidence The README and docs/local-llms.md said one contributor had run a local model and no transcript was in the repository. Two are now, under docs/assets/transcripts/: seed 6, where the previous run's episode sits in the system prompt under "What you remember" and the model never calls `remember`, and seed 5 after `remember` became strategy step 5, where it calls it once with a fact from the verb results. Both successes, every turn a native tool call, no text fallback, 14 s and 11 s of wall clock. Only the contributor's home directory was removed from `duck_path`. docs/local-llms.md reads them in a table and says what they cannot show; docs/assets/README.md records how they were made; the README's two sentences point at the folder; CHANGELOG under Unreleased. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 7 ++ README.md | 4 +- docs/assets/README.md | 2 + ...mstudio-find-and-kick-seed5-remember.jsonl | 79 +++++++++++++ ...udio-find-and-kick-seed6-memory-read.jsonl | 109 ++++++++++++++++++ docs/local-llms.md | 23 ++-- 6 files changed, 214 insertions(+), 10 deletions(-) create mode 100644 docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl create mode 100644 docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl diff --git a/CHANGELOG.md b/CHANGELOG.md index 0241cf3..b41f3c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Local model evidence + +- Two transcripts of `find-and-kick` piloted by **Qwen 2.5 Coder 14B on LM Studio**, seeds + 5 and 6, land in `docs/assets/transcripts/` with a table reading them in + `docs/local-llms.md`, from the contributor whose memory feature they were recorded for. + The README and `local-llms.md` no longer say "no transcript in this repository". + Two hardware paths, each audited against upstream rather than against itself. Still nothing has run on a robot; what changed is that several things which could not have worked now can, and several claims that were not true no longer are. diff --git a/README.md b/README.md index 5602f98..23ecb3e 100644 --- a/README.md +++ b/README.md @@ -216,7 +216,7 @@ The eight `duck_*` tools from 0.3 were aliases of the default robot and were rem | MCP server (`quackd serve-mcp`) | ✅ Claude Code and Claude Desktop, the Claude Code config checked against its docs (2026-08), no Claude Desktop session on record, fleets with `--robots` (eight `robot_*` tools, tested in process against the simulator and the mocks) | | Memory between runs (`quackd memory`, `remember`) | ✅ one JSONL file per `adapter:backend`, notes and run outcomes into the next prompt, tested end to end offline, 🧪 the `remember` tool itself exercised by one local model on one machine and by no cloud model ([docs/memory.md](docs/memory.md)) | | Providers: anthropic, openai, gemini, grok, fake | ✅ implemented, tested offline, real model hero recording pending an API key | -| Local models (Ollama, vLLM, llama.cpp, LM Studio, any OpenAI compatible server) | ✅ implemented and tested against the OpenAI wire format, 🧪 one live run by a contributor (Qwen 2.5 Coder 14B on LM Studio, two seeds), never on this machine and no transcript in the repo, more welcome | +| Local models (Ollama, vLLM, llama.cpp, LM Studio, any OpenAI compatible server) | ✅ implemented and tested against the OpenAI wire format, 🧪 two live runs by a contributor (Qwen 2.5 Coder 14B on LM Studio, seeds 5 and 6), never on this machine, transcripts in [`docs/assets/transcripts/`](docs/assets/transcripts/), more welcome | | Flock mode (multiple cooperating robots, sim2d) | ✅ deterministic auction and bus, one planner LLM call at most, ground truth checked in tests, 🧪 experimental and simulator only | | Real Microduck over JSON RPC (`--robot microduck:jsonrpc`) | 🧪 experimental, method names verified against upstream `duck-ipc-proto` v23, never run on hardware | | WebSocket agent gateway (`--robot microduck:websocket`) | ⏳ stub tracking upstream's draft ([architecture.md §5.3](https://github.com/pollen-robotics/microduck/blob/main/docs/design/architecture.md)) | @@ -569,7 +569,7 @@ Measured on the simulator with the scripted pilot (no model latency): `find-and- - `grab` is open loop upstream and unreliable here on purpose. `fetch` says so in its file. - A manifest can be smaller than the robot. The LeRobot arm's `real` backend claims no camera and no `pick` until it connects, and even then `pick` appears only when a policy object was injected in code, never from the command line. A rosbridge base over `ws` has no camera verbs unless the address names an image topic. - Default model IDs for OpenAI, Gemini and Grok were not verified at release. -- Local model quality is unmeasured. The JSON text fallback and the one retry exist because small models often miss native tool calls. No live local server has been run on this machine. One contributor reports running `find-and-kick` against Qwen 2.5 Coder 14B through LM Studio while building memory between runs, on two seeds, both successes with memory read and written, and no transcript from it is in this repository. +- Local model quality is unmeasured. The JSON text fallback and the one retry exist because small models often miss native tool calls. No live local server has been run on this machine. One contributor ran `find-and-kick` against Qwen 2.5 Coder 14B through LM Studio while building memory between runs, on two seeds, both successes with memory read and written. The two transcripts are in [`docs/assets/transcripts/`](docs/assets/transcripts/) and read in [docs/local-llms.md](docs/local-llms.md). - Flock mode is simulator only, and two choreographies ship today (`flock-kick`, where the closest duck acts on a target, and `reachy-spots-duck-kicks`, where a head spots and judges and a duck kicks), and quackd knows exactly two roles, spotter and kicker. Flock mode also knows only the Microduck and the Reachy Mini, so an Open Duck cannot join one yet. The coordination machinery is general, the choreography library is not, yet. The per robot pilots are deterministic rules, the LLM contributes one planning call at most, separation uses sim ground truth, not perception, and two robots share no frame of reference on hardware: the spotter judges from its own frames, and the arena frame hints that choose the kicker's first turn exist only in the simulator. - LAN discovery and the MQTT bus have each been exercised once, on one machine. Nothing has crossed to a second machine, the MQTT bus is a library with no `--bus` flag, and a flock across machines also needs a clock across machines, which does not exist yet. diff --git a/docs/assets/README.md b/docs/assets/README.md index d3caf42..a86d4c0 100644 --- a/docs/assets/README.md +++ b/docs/assets/README.md @@ -9,6 +9,8 @@ | `flock.gif` | Flock mode: three ducks split the search, auction the kick, the closest one takes it. Left: the shared world. Right: the claimant's camera. | `quackd record flock-kick --provider fake --seed 3` — the **scripted planner and deterministic coordinator**, no LLM in the loop (docs/flock.md). | | `hetero.gif` | The 0.4 headline demo: a Reachy Mini head (the slate square on the wall) spots the ball and judges the kick from its own camera while a Microduck walks in and kicks it, two bodies under one contract. Left: the shared world. Right: the acting robot's camera. | `quackd run reachy-spots-duck-kicks --provider fake --seed 3 --gif-size 320` — **scripted pilots and a deterministic coordinator**, zero planner LLM calls. That run ended `success`: the spotter judged the ball moved 0.51 m and the simulator's ground truth agreed. `quackd record` is pinned to `microduck:sim2d`, so this one comes from `run`, which also writes `run.gif`. | | `open-duck.gif` | The 0.5 headline body: an Open Duck Mini v2 finds the ball and walks up to it, because it has no kick. Left: the world from above. Right: the duck's camera. | `quackd run open-duck-scout --robot open_duck:sim2d --provider fake --seed 3 --gif-size 320` — the **scripted pilot**, not an LLM, like every other asset here. | +| `transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl` | A real local model piloting `find-and-kick`: Qwen 2.5 Coder 14B through LM Studio, seed 6, success in 8 steps, with the previous run's episode in the prompt and no `remember` call. | `quackd run find-and-kick --provider lmstudio --model qwen/qwen2.5-coder-14b --seed 6`, copied from `runs//transcript.jsonl` with the contributor's home directory removed from `duck_path`. Read in [docs/local-llms.md](../local-llms.md). | +| `transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl` | The same model and duck once `remember` sat in strategy step 5: seed 5, success in 4 steps, one `remember` call with a fact from the verb results. | Same command with `--seed 5`, same scrubbing. | | `contributors.svg` | The circle of faces under *Contributing* in the README: everyone who has contributed, humans only, ordered by lines added, every avatar clipped to the same circle. | Generated by [`contributors.py`](contributors.py) from the GitHub contributors API, avatars inlined so the file needs no network. Regenerated by `.github/workflows/contributors.yml` on every push to `main` and weekly, committed only when it changes. Two hosted services were tried first: one was three days stale, the other could not exclude bots or normalise a non square avatar. | | *(missing)* `reachy-spotter.gif` | **Not recorded yet.** A head with no legs finding the ball with its gaze, the simplest illustration that the verb list follows the body. | Would be `quackd run reachy-spotter --provider fake --seed 3`. | diff --git a/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl b/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl new file mode 100644 index 0000000..0257b05 --- /dev/null +++ b/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl @@ -0,0 +1,79 @@ +{"t": 0.005, "kind": "run_start", "duck": "find-and-kick", "duck_path": "ducks/find-and-kick.duck", "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "transport": "sim2d", "adapter": "microduck", "robot": {"manifest": 1, "id": "microduck", "vendor": "pollen-robotics", "model": "microduck", "embodiment": "biped", "mobility": "legged", "intents": ["twist", "skill", "gaze", "sound", "pose"], "sensors": ["camera", "battery", "odometry", "imu", "tof"], "verbs": [{"name": "observe", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "report_state", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stop", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "say", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {"text": {"description": "What to say.", "maxLength": 200, "minLength": 1, "title": "Text", "type": "string"}}, "required": ["text"], "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "move", "core": true, "description": "Walk with a velocity for a duration. Use small values; the robot is 25 cm tall.", "params_schema": {"additionalProperties": false, "properties": {"vx": {"default": 0.15, "description": "Forward m/s (negative = back).", "maximum": 0.3, "minimum": -0.3, "title": "Vx", "type": "number"}, "vy": {"default": 0.0, "description": "Left m/s (negative = right).", "maximum": 0.2, "minimum": -0.2, "title": "Vy", "type": "number"}, "wz": {"default": 0.0, "description": "Turn rate rad/s (+ = left).", "maximum": 1.5, "minimum": -1.5, "title": "Wz", "type": "number"}, "duration_s": {"default": 1.0, "description": "How long to hold this velocity.", "exclusiveMinimum": 0, "maximum": 10, "title": "Duration S", "type": "number"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "go_to", "core": true, "description": "Walk toward a detected target and stop at a distance. Closes the loop on the camera itself.", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "description": "Label to approach.", "title": "Target", "type": "string"}, "stop_distance": {"default": 0.25, "description": "Stop this far away (m).", "maximum": 1.5, "minimum": 0.1, "title": "Stop Distance", "type": "number"}, "timeout_s": {"default": 20.0, "exclusiveMinimum": 0, "maximum": 60, "title": "Timeout S", "type": "number"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "search_scan", "core": true, "description": "Rotate in steps, looking for a target. Returns where it was seen.", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "description": "Label to look for (e.g. ball, person).", "title": "Target", "type": "string"}, "step_deg": {"default": 45.0, "description": "Rotation per step.", "maximum": 120, "minimum": 15, "title": "Step Deg", "type": "number"}, "max_steps": {"default": 8, "description": "Steps before giving up (8 x 45° = full turn).", "maximum": 16, "minimum": 1, "title": "Max Steps", "type": "integer"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "approach_and", "core": true, "description": "walk_to a target, then run another verb (kick, grab).", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "title": "Target", "type": "string"}, "stop_distance": {"default": 0.25, "maximum": 1.5, "minimum": 0.1, "title": "Stop Distance", "type": "number"}, "then": {"description": "Verb to run once close (e.g. kick, grab).", "title": "Then", "type": "string"}}, "required": ["then"], "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "sit", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stand", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "kick", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"leg": {"default": "right", "description": "Which leg kicks.", "enum": ["left", "right"], "title": "Leg", "type": "string"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "grab", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stand_up", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "quack", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"text": {"anyOf": [{"maxLength": 200, "type": "string"}, {"type": "null"}], "default": null, "description": "What to say. The robot only has duck sounds; text is mapped to a tone.", "title": "Text"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "gaze", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"direction": {"default": "center", "description": "Where to point the head.", "enum": ["center", "left", "right", "up", "down"], "title": "Direction", "type": "string"}, "bearing_deg": {"anyOf": [{"maximum": 90, "minimum": -90, "type": "number"}, {"type": "null"}], "default": null, "description": "Optional exact bearing (+ = left). Overrides direction.", "title": "Bearing Deg"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}], "preconditions": {"move": ["standing"], "kick": ["standing"], "grab": ["standing"], "sit": ["not_fallen"], "stand": ["not_fallen"], "gaze": ["not_fallen"]}, "safety_authority": {"native": "robotd_deadman", "deadman": true, "heartbeat_hz": 2.0}, "frame": {"reference": "body", "note": ""}, "limits": {"max_vx": 0.3, "max_vy": 0.2, "max_wz": 1.5}, "backend": "sim2d", "blurb": "a small biped duck robot (25 cm, 800 g)", "extras": {}}, "dry_run": false, "contract": {"duck": 0, "name": "find-and-kick", "description": "Search the area for a ball, walk to it, kick it.", "author": "rok", "verbs": {"allow": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"], "confirm": []}, "budgets": {"max_steps": 40, "max_minutes": 5.0, "max_llm_calls": 40}, "success": ["Ball displaced more than 0.3 m in sim, or human confirms the kick landed."], "abort_when": ["Battery below 15%", "Same verb fails 3 times in a row"], "persona": "Determined and cheerful. Quack once when you succeed.", "providers": ["fake", "anthropic", "openai", "gemini", "grok"], "learned_verbs": [], "flock": null, "requires": [], "robots": null}, "system_prompt": "You are the brain of a small biped duck robot (25 cm, 800 g). You are a high-level pilot:\nyou choose ONE verb per turn; the robot's own controllers handle balance and gait, and composite\nverbs like `walk_to` close their own loops on the camera. Do not micro-manage.\n\n## Rules (enforced by the executor — not optional)\n- Call exactly one tool per turn. Never zero, never two.\n- Only these verbs are allowed: search_scan, walk_to, kick, quack, get_frame, stop. Anything else is refused.\n- Budgets: 40 steps, 5 minutes, 40 LLM calls. The run stops when any is hit.\n- Verbs marked confirm (none) ask a human before running.\n- When a success criterion is met, call `declare_success`. If the task is impossible, call `declare_failure`.\n\n## Success criteria\n- Ball displaced more than 0.3 m in sim, or human confirms the kick landed.\n\n## Abort conditions you must respect yourself\n- (none beyond the enforced ones)\n\n## Verbs\n- `search_scan`: Rotate in steps, looking for a target. Returns where it was seen.\n- `walk_to`: Walk toward a detected target and stop at a distance. Closes the loop on the camera itself.\n- `kick`: Kick forward with one leg. Only connects if the ball is < 0.3 m away and roughly ahead.\n- `quack`: Make a duck sound, optionally with text (mapped to a tone).\n- `get_frame`: Capture a camera frame and report what is detected in it.\n- `stop`: Stop moving immediately (zero velocity). Always allowed.\n\n## Persona\nDetermined and cheerful. Quack once when you succeed.\n\n## What you remember from earlier runs on this robot\n(nothing yet — this is the first run on this robot)\n\nCall `remember` (one short sentence) when you learn something worth keeping for next time:\nwhere an object usually is, which strategy worked, what to avoid. It is free: it moves\nnothing and costs no step. Do not save what is already listed above.\n\nYou are in the built-in 2D simulator: a cartoon top-down world. Distances are metres, the arena is about 2 m across, and the ball is orange.\n\n## Task file: find-and-kick — Search the area for a ball, walk to it, kick it.\n\n# Task\n\nYou are piloting a small biped duck robot. Find the ball and kick it.\n\n## Strategy\n\n1. `search_scan` to locate the ball. It rotates in steps and reports detections.\n2. `walk_to` the ball and stop about 0.25 m away. `walk_to` closes the approach loop\n itself using the camera — you do not need to micro-manage it.\n3. `kick`. The result tells you whether the ball moved and by how much.\n4. Verify with a fresh frame or the kick result. If the ball has not moved at least\n 0.3 m, `walk_to` it again and retry the kick.\n5. When the ball has moved ≥ 0.3 m, `remember` where you found the ball, `quack` once\n and declare success.\n\n## Notes\n\n- A kick only connects if the ball is close (< 0.3 m) and roughly in front of you.\n If the kick reports \"missed\", reposition with `walk_to` before kicking again.\n- If `search_scan` finds nothing twice in a row, declare failure rather than looping.\n\n## Memory\n\nBefore you declare success or failure, call `remember` once with one short fact a future\nrun can use: where the ball was when you found it (bearing and rough distance from where\nyou started) or what made the kick connect or miss. Skip it if the prompt already\nremembers the same thing. It costs no step.\n\n## If you cannot call tools natively\nReply with exactly one JSON object and nothing else, in this shape:\n{\"name\": \"\", \"arguments\": {}}\n", "tools": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop", "declare_success", "declare_failure", "remember"], "memory": {"robot": "microduck:sim2d", "path": "../memory-qwen3/microduck-sim2d.jsonl", "notes": 0, "episodes": 0}} +{"t": 0.018, "kind": "frame", "path": "frames/0000.png", "caption": "step 0: start"} +{"t": 0.019, "kind": "observation", "step": 0, "text": "[step 0/40 · step 0/40, llm calls 0/40, 0.0/5 min]\nstate: posture=standing policy=stand battery=100% pose=(0.18, 0.18, 0.10 rad)\ncamera: nothing detected\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 0.0, "x": 0.18300175424722812, "y": 0.18476447384189623, "theta": 0.09629333996427047, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 100.0, "holding": false, "extras": {"sim_time": 0.0, "ball": {"x": -0.321, "y": -0.669, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [], "last_result": null, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 4.492, "kind": "llm", "step": 0, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "902543897", "name": "search_scan", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 2551, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 4.493, "kind": "frame", "path": "frames/0001.png", "caption": "search_scan 0"} +{"t": 4.501, "kind": "frame", "path": "frames/0002.png", "caption": "search_scan 1"} +{"t": 4.512, "kind": "frame", "path": "frames/0003.png", "caption": "search_scan 2"} +{"t": 4.52, "kind": "frame", "path": "frames/0004.png", "caption": "search_scan 3"} +{"t": 4.527, "kind": "frame", "path": "frames/0005.png", "caption": "search_scan 4"} +{"t": 4.535, "kind": "frame", "path": "frames/0006.png", "caption": "search_scan 5"} +{"t": 4.537, "kind": "verb", "step": 1, "name": "search_scan", "canonical": "search_scan", "params": {"target": "ball"}, "ok": true, "summary": "ball found: ball at bearing 5° left ~1.00 m (after 5 turn steps)", "data": {"detections": [{"label": "ball", "cx": 0.453125, "cy": 0.5234375, "area": 0.0019683837890625, "confidence": 0.8225, "bearing_deg": 5.4, "est_distance_m": 0.999}], "steps": 5}} +{"t": 4.539, "kind": "frame", "path": "frames/0007.png", "caption": "step 1: search_scan"} +{"t": 4.542, "kind": "observation", "step": 1, "text": "[step 1/40 · step 1/40, llm calls 1/40, 0.1/5 min]\nstate: posture=standing policy=stand battery=100% pose=(0.18, 0.18, -2.20 rad)\ncamera: ball at bearing 5° left ~1.00 m; person at bearing 9° right ~1.23 m\nlast verb `search_scan`: ok — ball found: ball at bearing 5° left ~1.00 m (after 5 turn steps)\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 4.499999999999992, "x": 0.18300175424722812, "y": 0.18476447384189623, "theta": -2.1962847618103494, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.91, "holding": false, "extras": {"sim_time": 4.5, "ball": {"x": -0.321, "y": -0.669, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.453125, "cy": 0.5234375, "area": 0.0019683837890625, "confidence": 0.8225, "bearing_deg": 5.4, "est_distance_m": 0.999}, {"label": "person", "cx": 0.580078125, "cy": 0.4296875, "area": 0.0179443359375, "confidence": 1.0, "bearing_deg": -9.1, "est_distance_m": 1.229}], "last_result": {"verb": "search_scan", "ok": true, "summary": "ball found: ball at bearing 5° left ~1.00 m (after 5 turn steps)", "data": {"detections": [{"label": "ball", "cx": 0.453125, "cy": 0.5234375, "area": 0.0019683837890625, "confidence": 0.8225, "bearing_deg": 5.4, "est_distance_m": 0.999}], "steps": 5}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 8.376, "kind": "llm", "step": 1, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "929129346", "name": "walk_to", "arguments": {"target": "ball", "stop_distance": 0.25}}], "usage": {"input_tokens": 2726, "output_tokens": 35}, "stop_reason": "tool_calls"} +{"t": 8.378, "kind": "frame", "path": "frames/0008.png", "caption": "go_to ball"} +{"t": 8.383, "kind": "frame", "path": "frames/0009.png", "caption": "go_to ball"} +{"t": 8.386, "kind": "frame", "path": "frames/0010.png", "caption": "go_to ball"} +{"t": 8.389, "kind": "frame", "path": "frames/0011.png", "caption": "go_to ball"} +{"t": 8.391, "kind": "frame", "path": "frames/0012.png", "caption": "go_to ball"} +{"t": 8.393, "kind": "frame", "path": "frames/0013.png", "caption": "go_to ball"} +{"t": 8.396, "kind": "frame", "path": "frames/0014.png", "caption": "go_to ball"} +{"t": 8.398, "kind": "frame", "path": "frames/0015.png", "caption": "go_to ball"} +{"t": 8.401, "kind": "frame", "path": "frames/0016.png", "caption": "go_to ball"} +{"t": 8.403, "kind": "frame", "path": "frames/0017.png", "caption": "go_to ball"} +{"t": 8.406, "kind": "frame", "path": "frames/0018.png", "caption": "go_to ball"} +{"t": 8.408, "kind": "frame", "path": "frames/0019.png", "caption": "go_to ball"} +{"t": 8.41, "kind": "frame", "path": "frames/0020.png", "caption": "go_to ball"} +{"t": 8.412, "kind": "frame", "path": "frames/0021.png", "caption": "go_to ball"} +{"t": 8.414, "kind": "frame", "path": "frames/0022.png", "caption": "go_to ball"} +{"t": 8.417, "kind": "frame", "path": "frames/0023.png", "caption": "go_to ball"} +{"t": 8.419, "kind": "frame", "path": "frames/0024.png", "caption": "go_to ball"} +{"t": 8.421, "kind": "frame", "path": "frames/0025.png", "caption": "go_to ball"} +{"t": 8.423, "kind": "frame", "path": "frames/0026.png", "caption": "go_to ball"} +{"t": 8.426, "kind": "frame", "path": "frames/0027.png", "caption": "go_to ball"} +{"t": 8.428, "kind": "frame", "path": "frames/0028.png", "caption": "go_to ball"} +{"t": 8.431, "kind": "frame", "path": "frames/0029.png", "caption": "go_to ball"} +{"t": 8.433, "kind": "frame", "path": "frames/0030.png", "caption": "go_to ball"} +{"t": 8.435, "kind": "frame", "path": "frames/0031.png", "caption": "go_to ball"} +{"t": 8.438, "kind": "frame", "path": "frames/0032.png", "caption": "go_to ball"} +{"t": 8.44, "kind": "frame", "path": "frames/0033.png", "caption": "go_to ball"} +{"t": 8.442, "kind": "frame", "path": "frames/0034.png", "caption": "go_to ball"} +{"t": 8.445, "kind": "frame", "path": "frames/0035.png", "caption": "go_to ball"} +{"t": 8.447, "kind": "frame", "path": "frames/0036.png", "caption": "go_to ball"} +{"t": 8.449, "kind": "frame", "path": "frames/0037.png", "caption": "go_to ball"} +{"t": 8.451, "kind": "frame", "path": "frames/0038.png", "caption": "go_to ball"} +{"t": 8.453, "kind": "frame", "path": "frames/0039.png", "caption": "go_to ball"} +{"t": 8.456, "kind": "frame", "path": "frames/0040.png", "caption": "go_to ball"} +{"t": 8.458, "kind": "frame", "path": "frames/0041.png", "caption": "go_to ball"} +{"t": 8.46, "kind": "frame", "path": "frames/0042.png", "caption": "go_to ball"} +{"t": 8.462, "kind": "frame", "path": "frames/0043.png", "caption": "go_to ball"} +{"t": 8.465, "kind": "frame", "path": "frames/0044.png", "caption": "go_to ball"} +{"t": 8.467, "kind": "frame", "path": "frames/0045.png", "caption": "go_to ball"} +{"t": 8.469, "kind": "frame", "path": "frames/0046.png", "caption": "go_to ball"} +{"t": 8.472, "kind": "frame", "path": "frames/0047.png", "caption": "go_to ball"} +{"t": 8.474, "kind": "frame", "path": "frames/0048.png", "caption": "go_to ball"} +{"t": 8.476, "kind": "frame", "path": "frames/0049.png", "caption": "go_to ball"} +{"t": 8.479, "kind": "frame", "path": "frames/0050.png", "caption": "go_to ball"} +{"t": 8.481, "kind": "frame", "path": "frames/0051.png", "caption": "go_to ball"} +{"t": 8.483, "kind": "frame", "path": "frames/0052.png", "caption": "go_to ball"} +{"t": 8.485, "kind": "frame", "path": "frames/0053.png", "caption": "go_to ball"} +{"t": 8.487, "kind": "verb", "step": 2, "name": "walk_to", "canonical": "go_to", "params": {"target": "ball", "stop_distance": 0.25}, "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.242, "bearing_deg": 0.0, "ticks": 46}} +{"t": 8.488, "kind": "frame", "path": "frames/0054.png", "caption": "step 2: walk_to"} +{"t": 8.489, "kind": "observation", "step": 2, "text": "[step 2/40 · step 2/40, llm calls 2/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.20, -0.46, -2.10 rad)\ncamera: ball dead ahead ~0.24 m; person at bearing 33° right ~0.58 m\nlast verb `walk_to`: ok — reached the ball: ~0.24 m away, bearing +0°\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 9.099999999999994, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.818, "holding": false, "extras": {"sim_time": 9.1, "ball": {"x": -0.321, "y": -0.669, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.5, "cy": 0.7578125, "area": 0.0335845947265625, "confidence": 1.0, "bearing_deg": -0.0, "est_distance_m": 0.242}, {"label": "person", "cx": 0.830078125, "cy": 0.40625, "area": 0.08648681640625, "confidence": 1.0, "bearing_deg": -33.4, "est_distance_m": 0.58}], "last_result": {"verb": "walk_to", "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.242, "bearing_deg": 0.0, "ticks": 46}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 11.63, "kind": "llm", "step": 2, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "380454315", "name": "kick", "arguments": {"leg": "right"}}], "usage": {"input_tokens": 2900, "output_tokens": 25}, "stop_reason": "tool_calls"} +{"t": 11.643, "kind": "verb", "step": 3, "name": "kick", "canonical": "kick", "params": {"leg": "right"}, "ok": true, "summary": "kicked with right leg, ball moved 0.38 m", "data": {"ball_moved_m": 0.379}} +{"t": 11.645, "kind": "frame", "path": "frames/0055.png", "caption": "step 3: kick"} +{"t": 11.646, "kind": "observation", "step": 3, "text": "[step 3/40 · step 3/40, llm calls 3/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.20, -0.46, -2.10 rad)\ncamera: ball at bearing 19° right ~0.61 m; person at bearing 33° right ~0.58 m\nlast verb `kick`: ok — kicked with right leg, ball moved 0.38 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 10.600000000000016, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.788, "holding": false, "extras": {"sim_time": 10.6, "ball": {"x": -0.652, "y": -0.854, "present": true}, "ball_displacement_m": 0.379, "last_kick_ball_moved_m": 0.379, "kicks": 1, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.66796875, "cy": 0.57421875, "area": 0.0052642822265625, "confidence": 1.0, "bearing_deg": -18.6, "est_distance_m": 0.611}, {"label": "person", "cx": 0.830078125, "cy": 0.40625, "area": 0.08648681640625, "confidence": 1.0, "bearing_deg": -33.4, "est_distance_m": 0.58}], "last_result": {"verb": "kick", "ok": true, "summary": "kicked with right leg, ball moved 0.38 m", "data": {"ball_moved_m": 0.379}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 20.794, "kind": "llm", "step": 3, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": "\n\n\n\n", "tool_calls": [{"id": "888225386", "name": "remember", "arguments": {"text": "the ball was found near the starting position and kicked 0.38 m to the right with the right leg"}}, {"id": "217792846", "name": "quack", "arguments": {}}, {"id": "628057790", "name": "declare_success", "arguments": {"reason": "the ball was successfully kicked 0.38 m to the right with the right leg"}}], "usage": {"input_tokens": 3066, "output_tokens": 111}, "stop_reason": "tool_calls"} +{"t": 20.794, "kind": "enforce", "step": 3, "issue": "multiple_tool_calls", "action": "first_only"} +{"t": 20.797, "kind": "memory", "step": 3, "ok": true, "text": "the ball was found near the starting position and kicked 0.38 m to the right with the right leg", "summary": "remembered for future runs: the ball was found near the starting position and kicked 0.38 m to the right with the right leg"} +{"t": 20.801, "kind": "frame", "path": "frames/0056.png", "caption": "step 3: remember"} +{"t": 20.804, "kind": "observation", "step": 3, "text": "[step 3/40 · step 3/40, llm calls 4/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.20, -0.46, -2.10 rad)\ncamera: ball at bearing 19° right ~0.61 m; person at bearing 33° right ~0.58 m\nlast verb `remember`: ok — remembered for future runs: the ball was found near the starting position and kicked 0.38 m to the right with the right leg\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 10.600000000000016, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.788, "holding": false, "extras": {"sim_time": 10.6, "ball": {"x": -0.652, "y": -0.854, "present": true}, "ball_displacement_m": 0.379, "last_kick_ball_moved_m": 0.379, "kicks": 1, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.66796875, "cy": 0.57421875, "area": 0.0052642822265625, "confidence": 1.0, "bearing_deg": -18.6, "est_distance_m": 0.611}, {"label": "person", "cx": 0.830078125, "cy": 0.40625, "area": 0.08648681640625, "confidence": 1.0, "bearing_deg": -33.4, "est_distance_m": 0.58}], "last_result": {"verb": "remember", "ok": true, "summary": "remembered for future runs: the ball was found near the starting position and kicked 0.38 m to the right with the right leg", "data": {"notes": 1}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 24.265, "kind": "llm", "step": 3, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "216390142", "name": "quack", "arguments": {}}], "usage": {"input_tokens": 3269, "output_tokens": 23}, "stop_reason": "tool_calls"} +{"t": 24.265, "kind": "verb", "step": 4, "name": "quack", "canonical": "quack", "params": {}, "ok": true, "summary": "quacked [chirp]", "data": {"tag": "chirp", "text": null}} +{"t": 24.267, "kind": "frame", "path": "frames/0057.png", "caption": "step 4: quack"} +{"t": 24.268, "kind": "observation", "step": 4, "text": "[step 4/40 · step 4/40, llm calls 5/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.20, -0.46, -2.10 rad)\ncamera: ball at bearing 19° right ~0.61 m; person at bearing 33° right ~0.58 m\nlast verb `quack`: ok — quacked [chirp]\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 10.600000000000016, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.788, "holding": false, "extras": {"sim_time": 10.6, "ball": {"x": -0.652, "y": -0.854, "present": true}, "ball_displacement_m": 0.379, "last_kick_ball_moved_m": 0.379, "kicks": 1, "quacks": 1, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.66796875, "cy": 0.57421875, "area": 0.0052642822265625, "confidence": 1.0, "bearing_deg": -18.6, "est_distance_m": 0.611}, {"label": "person", "cx": 0.830078125, "cy": 0.40625, "area": 0.08648681640625, "confidence": 1.0, "bearing_deg": -33.4, "est_distance_m": 0.58}], "last_result": {"verb": "quack", "ok": true, "summary": "quacked [chirp]", "data": {"tag": "chirp", "text": null}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 28.907, "kind": "llm", "step": 4, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "986592274", "name": "declare_success", "arguments": {"reason": "the ball was kicked successfully, moving 0.38 m to the right with the right leg"}}], "usage": {"input_tokens": 3427, "output_tokens": 45}, "stop_reason": "tool_calls"} +{"t": 28.907, "kind": "declare", "step": 4, "outcome": "success", "reason": "the ball was kicked successfully, moving 0.38 m to the right with the right leg"} +{"t": 28.908, "kind": "run_end", "duck": "find-and-kick", "outcome": "success", "reason": "the ball was kicked successfully, moving 0.38 m to the right with the right leg", "steps": 4, "llm_calls": 6, "elapsed_s": 10.6, "usage": {"input_tokens": 17939, "output_tokens": 265}, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "transport": "sim2d", "robot": "microduck", "dry_run": false, "final_state": {"t": 10.600000000000016, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.788, "holding": false, "extras": {"sim_time": 10.6, "ball": {"x": -0.652, "y": -0.854, "present": true}, "ball_displacement_m": 0.379, "last_kick_ball_moved_m": 0.379, "kicks": 1, "quacks": 1, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}} diff --git a/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl b/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl new file mode 100644 index 0000000..a00c2b7 --- /dev/null +++ b/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl @@ -0,0 +1,109 @@ +{"t": 0.006, "kind": "run_start", "duck": "find-and-kick", "duck_path": "ducks/find-and-kick.duck", "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "transport": "sim2d", "adapter": "microduck", "robot": {"manifest": 1, "id": "microduck", "vendor": "pollen-robotics", "model": "microduck", "embodiment": "biped", "mobility": "legged", "intents": ["twist", "skill", "gaze", "sound", "pose"], "sensors": ["camera", "battery", "odometry", "imu", "tof"], "verbs": [{"name": "observe", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "report_state", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stop", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "say", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {"text": {"description": "What to say.", "maxLength": 200, "minLength": 1, "title": "Text", "type": "string"}}, "required": ["text"], "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "move", "core": true, "description": "Walk with a velocity for a duration. Use small values; the robot is 25 cm tall.", "params_schema": {"additionalProperties": false, "properties": {"vx": {"default": 0.15, "description": "Forward m/s (negative = back).", "maximum": 0.3, "minimum": -0.3, "title": "Vx", "type": "number"}, "vy": {"default": 0.0, "description": "Left m/s (negative = right).", "maximum": 0.2, "minimum": -0.2, "title": "Vy", "type": "number"}, "wz": {"default": 0.0, "description": "Turn rate rad/s (+ = left).", "maximum": 1.5, "minimum": -1.5, "title": "Wz", "type": "number"}, "duration_s": {"default": 1.0, "description": "How long to hold this velocity.", "exclusiveMinimum": 0, "maximum": 10, "title": "Duration S", "type": "number"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "go_to", "core": true, "description": "Walk toward a detected target and stop at a distance. Closes the loop on the camera itself.", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "description": "Label to approach.", "title": "Target", "type": "string"}, "stop_distance": {"default": 0.25, "description": "Stop this far away (m).", "maximum": 1.5, "minimum": 0.1, "title": "Stop Distance", "type": "number"}, "timeout_s": {"default": 20.0, "exclusiveMinimum": 0, "maximum": 60, "title": "Timeout S", "type": "number"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "search_scan", "core": true, "description": "Rotate in steps, looking for a target. Returns where it was seen.", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "description": "Label to look for (e.g. ball, person).", "title": "Target", "type": "string"}, "step_deg": {"default": 45.0, "description": "Rotation per step.", "maximum": 120, "minimum": 15, "title": "Step Deg", "type": "number"}, "max_steps": {"default": 8, "description": "Steps before giving up (8 x 45° = full turn).", "maximum": 16, "minimum": 1, "title": "Max Steps", "type": "integer"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "approach_and", "core": true, "description": "walk_to a target, then run another verb (kick, grab).", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "title": "Target", "type": "string"}, "stop_distance": {"default": 0.25, "maximum": 1.5, "minimum": 0.1, "title": "Stop Distance", "type": "number"}, "then": {"description": "Verb to run once close (e.g. kick, grab).", "title": "Then", "type": "string"}}, "required": ["then"], "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "sit", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stand", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "kick", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"leg": {"default": "right", "description": "Which leg kicks.", "enum": ["left", "right"], "title": "Leg", "type": "string"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "grab", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stand_up", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "quack", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"text": {"anyOf": [{"maxLength": 200, "type": "string"}, {"type": "null"}], "default": null, "description": "What to say. The robot only has duck sounds; text is mapped to a tone.", "title": "Text"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "gaze", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"direction": {"default": "center", "description": "Where to point the head.", "enum": ["center", "left", "right", "up", "down"], "title": "Direction", "type": "string"}, "bearing_deg": {"anyOf": [{"maximum": 90, "minimum": -90, "type": "number"}, {"type": "null"}], "default": null, "description": "Optional exact bearing (+ = left). Overrides direction.", "title": "Bearing Deg"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}], "preconditions": {"move": ["standing"], "kick": ["standing"], "grab": ["standing"], "sit": ["not_fallen"], "stand": ["not_fallen"], "gaze": ["not_fallen"]}, "safety_authority": {"native": "robotd_deadman", "deadman": true, "heartbeat_hz": 2.0}, "frame": {"reference": "body", "note": ""}, "limits": {"max_vx": 0.3, "max_vy": 0.2, "max_wz": 1.5}, "backend": "sim2d", "blurb": "a small biped duck robot (25 cm, 800 g)", "extras": {}}, "dry_run": false, "contract": {"duck": 0, "name": "find-and-kick", "description": "Search the area for a ball, walk to it, kick it.", "author": "rok", "verbs": {"allow": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"], "confirm": []}, "budgets": {"max_steps": 40, "max_minutes": 5.0, "max_llm_calls": 40}, "success": ["Ball displaced more than 0.3 m in sim, or human confirms the kick landed."], "abort_when": ["Battery below 15%", "Same verb fails 3 times in a row"], "persona": "Determined and cheerful. Quack once when you succeed.", "providers": ["fake", "anthropic", "openai", "gemini", "grok"], "learned_verbs": [], "flock": null, "requires": [], "robots": null}, "system_prompt": "You are the brain of a small biped duck robot (25 cm, 800 g). You are a high-level pilot:\nyou choose ONE verb per turn; the robot's own controllers handle balance and gait, and composite\nverbs like `walk_to` close their own loops on the camera. Do not micro-manage.\n\n## Rules (enforced by the executor — not optional)\n- Call exactly one tool per turn. Never zero, never two.\n- Only these verbs are allowed: search_scan, walk_to, kick, quack, get_frame, stop. Anything else is refused.\n- Budgets: 40 steps, 5 minutes, 40 LLM calls. The run stops when any is hit.\n- Verbs marked confirm (none) ask a human before running.\n- When a success criterion is met, call `declare_success`. If the task is impossible, call `declare_failure`.\n\n## Success criteria\n- Ball displaced more than 0.3 m in sim, or human confirms the kick landed.\n\n## Abort conditions you must respect yourself\n- (none beyond the enforced ones)\n\n## Verbs\n- `search_scan`: Rotate in steps, looking for a target. Returns where it was seen.\n- `walk_to`: Walk toward a detected target and stop at a distance. Closes the loop on the camera itself.\n- `kick`: Kick forward with one leg. Only connects if the ball is < 0.3 m away and roughly ahead.\n- `quack`: Make a duck sound, optionally with text (mapped to a tone).\n- `get_frame`: Capture a camera frame and report what is detected in it.\n- `stop`: Stop moving immediately (zero velocity). Always allowed.\n\n## Persona\nDetermined and cheerful. Quack once when you succeed.\n\n## What you remember from earlier runs on this robot\nYour most recent runs on this robot (newest first):\n- [2026-09-03] find-and-kick: success — Ball moved 0.38 m after kick, meeting the success criteria. (4 steps) · search_scan: ball found: ball at bearing 5° left ~1.00 m (after 5 turn steps); walk_to: reached the ball: ~0.24 m away, bearing +0°; kick: kicked with right leg, ball moved 0.38 m; quack: quacked [chirp]\n\nCall `remember` (one short sentence) when you learn something worth keeping for next time:\nwhere an object usually is, which strategy worked, what to avoid. It is free: it moves\nnothing and costs no step. Do not save what is already listed above.\n\nYou are in the built-in 2D simulator: a cartoon top-down world. Distances are metres, the arena is about 2 m across, and the ball is orange.\n\n## Task file: find-and-kick — Search the area for a ball, walk to it, kick it.\n\n# Task\n\nYou are piloting a small biped duck robot. Find the ball and kick it.\n\n## Strategy\n\n1. `search_scan` to locate the ball. It rotates in steps and reports detections.\n2. `walk_to` the ball and stop about 0.25 m away. `walk_to` closes the approach loop\n itself using the camera — you do not need to micro-manage it.\n3. `kick`. The result tells you whether the ball moved and by how much.\n4. Verify with a fresh frame or the kick result. If the ball has not moved at least\n 0.3 m, `walk_to` it again and retry the kick.\n5. When the ball has moved ≥ 0.3 m, `quack` once and declare success.\n\n## Notes\n\n- A kick only connects if the ball is close (< 0.3 m) and roughly in front of you.\n If the kick reports \"missed\", reposition with `walk_to` before kicking again.\n- If `search_scan` finds nothing twice in a row, declare failure rather than looping.\n\n## Memory\n\nBefore you declare success or failure, call `remember` once with one short fact a future\nrun can use: where the ball was when you found it (bearing and rough distance from where\nyou started) or what made the kick connect or miss. Skip it if the prompt already\nremembers the same thing. It costs no step.\n\n## If you cannot call tools natively\nReply with exactly one JSON object and nothing else, in this shape:\n{\"name\": \"\", \"arguments\": {}}\n", "tools": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop", "declare_success", "declare_failure", "remember"], "memory": {"robot": "microduck:sim2d", "path": "../memory-qwen2/microduck-sim2d.jsonl", "notes": 0, "episodes": 1}} +{"t": 0.018, "kind": "frame", "path": "frames/0000.png", "caption": "step 0: start"} +{"t": 0.019, "kind": "observation", "step": 0, "text": "[step 0/40 · step 0/40, llm calls 0/40, 0.0/5 min]\nstate: posture=standing policy=stand battery=100% pose=(0.02, -0.09, -0.82 rad)\ncamera: nothing detected\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 0.0, "x": 0.022898610883165915, "y": -0.09403747811199693, "theta": -0.8226747951461474, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 100.0, "holding": false, "extras": {"sim_time": 0.0, "ball": {"x": -0.188, "y": 0.731, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [], "last_result": null, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 6.924, "kind": "llm", "step": 0, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "708275807", "name": "search_scan", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 2652, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 6.926, "kind": "frame", "path": "frames/0001.png", "caption": "search_scan 0"} +{"t": 6.933, "kind": "frame", "path": "frames/0002.png", "caption": "search_scan 1"} +{"t": 6.939, "kind": "frame", "path": "frames/0003.png", "caption": "search_scan 2"} +{"t": 6.945, "kind": "frame", "path": "frames/0004.png", "caption": "search_scan 3"} +{"t": 6.947, "kind": "verb", "step": 1, "name": "search_scan", "canonical": "search_scan", "params": {"target": "ball"}, "ok": true, "summary": "ball found: ball at bearing 14° left ~0.83 m (after 3 turn steps)", "data": {"detections": [{"label": "ball", "cx": 0.376953125, "cy": 0.5390625, "area": 0.002838134765625, "confidence": 0.9650000000000001, "bearing_deg": 13.8, "est_distance_m": 0.832}], "steps": 3}} +{"t": 6.948, "kind": "frame", "path": "frames/0005.png", "caption": "step 1: search_scan"} +{"t": 6.95, "kind": "observation", "step": 1, "text": "[step 1/40 · step 1/40, llm calls 1/40, 0.0/5 min]\nstate: posture=standing policy=stand battery=100% pose=(0.02, -0.09, 1.58 rad)\ncamera: ball at bearing 14° left ~0.83 m\nlast verb `search_scan`: ok — ball found: ball at bearing 14° left ~0.83 m (after 3 turn steps)\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 2.6999999999999984, "x": 0.022898610883165915, "y": -0.09403747811199693, "theta": 1.5799555297252343, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.946, "holding": false, "extras": {"sim_time": 2.7, "ball": {"x": -0.188, "y": 0.731, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.376953125, "cy": 0.5390625, "area": 0.002838134765625, "confidence": 0.9650000000000001, "bearing_deg": 13.8, "est_distance_m": 0.832}], "last_result": {"verb": "search_scan", "ok": true, "summary": "ball found: ball at bearing 14° left ~0.83 m (after 3 turn steps)", "data": {"detections": [{"label": "ball", "cx": 0.376953125, "cy": 0.5390625, "area": 0.002838134765625, "confidence": 0.9650000000000001, "bearing_deg": 13.8, "est_distance_m": 0.832}], "steps": 3}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 10.167, "kind": "llm", "step": 1, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "421996211", "name": "walk_to", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 2815, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 10.169, "kind": "frame", "path": "frames/0006.png", "caption": "go_to ball"} +{"t": 10.173, "kind": "frame", "path": "frames/0007.png", "caption": "go_to ball"} +{"t": 10.176, "kind": "frame", "path": "frames/0008.png", "caption": "go_to ball"} +{"t": 10.178, "kind": "frame", "path": "frames/0009.png", "caption": "go_to ball"} +{"t": 10.181, "kind": "frame", "path": "frames/0010.png", "caption": "go_to ball"} +{"t": 10.183, "kind": "frame", "path": "frames/0011.png", "caption": "go_to ball"} +{"t": 10.185, "kind": "frame", "path": "frames/0012.png", "caption": "go_to ball"} +{"t": 10.187, "kind": "frame", "path": "frames/0013.png", "caption": "go_to ball"} +{"t": 10.19, "kind": "frame", "path": "frames/0014.png", "caption": "go_to ball"} +{"t": 10.193, "kind": "frame", "path": "frames/0015.png", "caption": "go_to ball"} +{"t": 10.195, "kind": "frame", "path": "frames/0016.png", "caption": "go_to ball"} +{"t": 10.197, "kind": "frame", "path": "frames/0017.png", "caption": "go_to ball"} +{"t": 10.2, "kind": "frame", "path": "frames/0018.png", "caption": "go_to ball"} +{"t": 10.202, "kind": "frame", "path": "frames/0019.png", "caption": "go_to ball"} +{"t": 10.204, "kind": "frame", "path": "frames/0020.png", "caption": "go_to ball"} +{"t": 10.207, "kind": "frame", "path": "frames/0021.png", "caption": "go_to ball"} +{"t": 10.209, "kind": "frame", "path": "frames/0022.png", "caption": "go_to ball"} +{"t": 10.211, "kind": "frame", "path": "frames/0023.png", "caption": "go_to ball"} +{"t": 10.213, "kind": "frame", "path": "frames/0024.png", "caption": "go_to ball"} +{"t": 10.215, "kind": "frame", "path": "frames/0025.png", "caption": "go_to ball"} +{"t": 10.218, "kind": "frame", "path": "frames/0026.png", "caption": "go_to ball"} +{"t": 10.22, "kind": "frame", "path": "frames/0027.png", "caption": "go_to ball"} +{"t": 10.222, "kind": "frame", "path": "frames/0028.png", "caption": "go_to ball"} +{"t": 10.224, "kind": "frame", "path": "frames/0029.png", "caption": "go_to ball"} +{"t": 10.227, "kind": "frame", "path": "frames/0030.png", "caption": "go_to ball"} +{"t": 10.229, "kind": "frame", "path": "frames/0031.png", "caption": "go_to ball"} +{"t": 10.231, "kind": "frame", "path": "frames/0032.png", "caption": "go_to ball"} +{"t": 10.233, "kind": "frame", "path": "frames/0033.png", "caption": "go_to ball"} +{"t": 10.235, "kind": "frame", "path": "frames/0034.png", "caption": "go_to ball"} +{"t": 10.237, "kind": "frame", "path": "frames/0035.png", "caption": "go_to ball"} +{"t": 10.24, "kind": "frame", "path": "frames/0036.png", "caption": "go_to ball"} +{"t": 10.242, "kind": "frame", "path": "frames/0037.png", "caption": "go_to ball"} +{"t": 10.244, "kind": "frame", "path": "frames/0038.png", "caption": "go_to ball"} +{"t": 10.246, "kind": "frame", "path": "frames/0039.png", "caption": "go_to ball"} +{"t": 10.248, "kind": "frame", "path": "frames/0040.png", "caption": "go_to ball"} +{"t": 10.251, "kind": "frame", "path": "frames/0041.png", "caption": "go_to ball"} +{"t": 10.253, "kind": "frame", "path": "frames/0042.png", "caption": "go_to ball"} +{"t": 10.255, "kind": "frame", "path": "frames/0043.png", "caption": "go_to ball"} +{"t": 10.257, "kind": "frame", "path": "frames/0044.png", "caption": "go_to ball"} +{"t": 10.259, "kind": "verb", "step": 2, "name": "walk_to", "canonical": "go_to", "params": {"target": "ball"}, "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.242, "bearing_deg": 0.0, "ticks": 39}} +{"t": 10.26, "kind": "frame", "path": "frames/0045.png", "caption": "step 2: walk_to"} +{"t": 10.261, "kind": "observation", "step": 2, "text": "[step 2/40 · step 2/40, llm calls 2/40, 0.1/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.12, 0.50, 1.84 rad)\ncamera: ball dead ahead ~0.24 m\nlast verb `walk_to`: ok — reached the ball: ~0.24 m away, bearing +0°\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 6.5999999999999845, "x": -0.1242526964459676, "y": 0.49863001147922587, "theta": 1.8404184250668225, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.868, "holding": false, "extras": {"sim_time": 6.6, "ball": {"x": -0.188, "y": 0.731, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.5, "cy": 0.76171875, "area": 0.0335845947265625, "confidence": 1.0, "bearing_deg": -0.0, "est_distance_m": 0.242}], "last_result": {"verb": "walk_to", "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.242, "bearing_deg": 0.0, "ticks": 39}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 13.217, "kind": "llm", "step": 2, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "401399688", "name": "kick", "arguments": {"leg": "right"}}], "usage": {"input_tokens": 2966, "output_tokens": 25}, "stop_reason": "tool_calls"} +{"t": 13.226, "kind": "verb", "step": 3, "name": "kick", "canonical": "kick", "params": {"leg": "right"}, "ok": true, "summary": "kicked with right leg, ball moved 0.15 m", "data": {"ball_moved_m": 0.152}} +{"t": 13.227, "kind": "frame", "path": "frames/0046.png", "caption": "step 3: kick"} +{"t": 13.229, "kind": "observation", "step": 3, "text": "[step 3/40 · step 3/40, llm calls 3/40, 0.1/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.12, 0.50, 1.84 rad)\ncamera: ball at bearing 10° left ~0.39 m\nlast verb `kick`: ok — kicked with right leg, ball moved 0.15 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 8.09999999999998, "x": -0.1242526964459676, "y": 0.49863001147922587, "theta": 1.8404184250668225, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.838, "holding": false, "extras": {"sim_time": 8.1, "ball": {"x": -0.29, "y": 0.844, "present": true}, "ball_displacement_m": 0.152, "last_kick_ball_moved_m": 0.152, "kicks": 1, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.41015625, "cy": 0.64453125, "area": 0.0130767822265625, "confidence": 1.0, "bearing_deg": 10.2, "est_distance_m": 0.387}], "last_result": {"verb": "kick", "ok": true, "summary": "kicked with right leg, ball moved 0.15 m", "data": {"ball_moved_m": 0.152}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 16.244, "kind": "llm", "step": 3, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "283888237", "name": "walk_to", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 3117, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 16.247, "kind": "frame", "path": "frames/0047.png", "caption": "go_to ball"} +{"t": 16.25, "kind": "frame", "path": "frames/0048.png", "caption": "go_to ball"} +{"t": 16.252, "kind": "frame", "path": "frames/0049.png", "caption": "go_to ball"} +{"t": 16.255, "kind": "frame", "path": "frames/0050.png", "caption": "go_to ball"} +{"t": 16.257, "kind": "frame", "path": "frames/0051.png", "caption": "go_to ball"} +{"t": 16.259, "kind": "frame", "path": "frames/0052.png", "caption": "go_to ball"} +{"t": 16.262, "kind": "frame", "path": "frames/0053.png", "caption": "go_to ball"} +{"t": 16.264, "kind": "frame", "path": "frames/0054.png", "caption": "go_to ball"} +{"t": 16.266, "kind": "frame", "path": "frames/0055.png", "caption": "go_to ball"} +{"t": 16.268, "kind": "frame", "path": "frames/0056.png", "caption": "go_to ball"} +{"t": 16.27, "kind": "frame", "path": "frames/0057.png", "caption": "go_to ball"} +{"t": 16.274, "kind": "frame", "path": "frames/0058.png", "caption": "go_to ball"} +{"t": 16.276, "kind": "frame", "path": "frames/0059.png", "caption": "go_to ball"} +{"t": 16.279, "kind": "frame", "path": "frames/0060.png", "caption": "go_to ball"} +{"t": 16.281, "kind": "frame", "path": "frames/0061.png", "caption": "go_to ball"} +{"t": 16.283, "kind": "verb", "step": 4, "name": "walk_to", "canonical": "go_to", "params": {"target": "ball"}, "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.244, "bearing_deg": 0.2, "ticks": 15}} +{"t": 16.284, "kind": "frame", "path": "frames/0062.png", "caption": "step 4: walk_to"} +{"t": 16.285, "kind": "observation", "step": 4, "text": "[step 4/40 · step 4/40, llm calls 4/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.18, 0.63, 2.03 rad)\ncamera: ball dead ahead ~0.24 m\nlast verb `walk_to`: ok — reached the ball: ~0.24 m away, bearing +0°\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 9.600000000000001, "x": -0.18148846702554064, "y": 0.6260237651230226, "theta": 2.0316710037782926, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.808, "holding": false, "extras": {"sim_time": 9.6, "ball": {"x": -0.29, "y": 0.844, "present": true}, "ball_displacement_m": 0.152, "last_kick_ball_moved_m": 0.152, "kicks": 1, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.498046875, "cy": 0.7578125, "area": 0.033111572265625, "confidence": 1.0, "bearing_deg": 0.2, "est_distance_m": 0.244}], "last_result": {"verb": "walk_to", "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.244, "bearing_deg": 0.2, "ticks": 15}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 19.264, "kind": "llm", "step": 4, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "961913944", "name": "kick", "arguments": {"leg": "right"}}], "usage": {"input_tokens": 3268, "output_tokens": 25}, "stop_reason": "tool_calls"} +{"t": 19.276, "kind": "verb", "step": 5, "name": "kick", "canonical": "kick", "params": {"leg": "right"}, "ok": true, "summary": "kicked with right leg, ball moved 0.18 m", "data": {"ball_moved_m": 0.183}} +{"t": 19.277, "kind": "frame", "path": "frames/0063.png", "caption": "step 5: kick"} +{"t": 19.278, "kind": "observation", "step": 5, "text": "[step 5/40 · step 5/40, llm calls 5/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.18, 0.63, 2.03 rad)\ncamera: ball at bearing 31° left ~0.34 m\nlast verb `kick`: ok — kicked with right leg, ball moved 0.18 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 11.100000000000023, "x": -0.18148846702554064, "y": 0.6260237651230226, "theta": 2.0316710037782926, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.778, "holding": false, "extras": {"sim_time": 11.1, "ball": {"x": -0.47, "y": 0.809, "present": true}, "ball_displacement_m": 0.293, "last_kick_ball_moved_m": 0.183, "kicks": 2, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.197265625, "cy": 0.66796875, "area": 0.01690673828125, "confidence": 1.0, "bearing_deg": 31.2, "est_distance_m": 0.341}], "last_result": {"verb": "kick", "ok": true, "summary": "kicked with right leg, ball moved 0.18 m", "data": {"ball_moved_m": 0.183}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 22.361, "kind": "llm", "step": 5, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "646573368", "name": "walk_to", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 3419, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 22.364, "kind": "frame", "path": "frames/0064.png", "caption": "go_to ball"} +{"t": 22.37, "kind": "frame", "path": "frames/0065.png", "caption": "go_to ball"} +{"t": 22.373, "kind": "frame", "path": "frames/0066.png", "caption": "go_to ball"} +{"t": 22.375, "kind": "frame", "path": "frames/0067.png", "caption": "go_to ball"} +{"t": 22.378, "kind": "frame", "path": "frames/0068.png", "caption": "go_to ball"} +{"t": 22.38, "kind": "frame", "path": "frames/0069.png", "caption": "go_to ball"} +{"t": 22.383, "kind": "frame", "path": "frames/0070.png", "caption": "go_to ball"} +{"t": 22.385, "kind": "frame", "path": "frames/0071.png", "caption": "go_to ball"} +{"t": 22.388, "kind": "frame", "path": "frames/0072.png", "caption": "go_to ball"} +{"t": 22.39, "kind": "frame", "path": "frames/0073.png", "caption": "go_to ball"} +{"t": 22.392, "kind": "frame", "path": "frames/0074.png", "caption": "go_to ball"} +{"t": 22.394, "kind": "frame", "path": "frames/0075.png", "caption": "go_to ball"} +{"t": 22.396, "kind": "verb", "step": 6, "name": "walk_to", "canonical": "go_to", "params": {"target": "ball"}, "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +1°", "data": {"distance_m": 0.244, "bearing_deg": 1.3, "ticks": 12}} +{"t": 22.397, "kind": "frame", "path": "frames/0076.png", "caption": "step 6: walk_to"} +{"t": 22.398, "kind": "observation", "step": 6, "text": "[step 6/40 · step 6/40, llm calls 6/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.26, 0.69, 2.60 rad)\ncamera: ball dead ahead ~0.24 m\nlast verb `walk_to`: ok — reached the ball: ~0.24 m away, bearing +1°\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 12.30000000000004, "x": -0.2585395167483543, "y": 0.688311967025222, "theta": 2.598946723755427, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.754, "holding": false, "extras": {"sim_time": 12.3, "ball": {"x": -0.47, "y": 0.809, "present": true}, "ball_displacement_m": 0.293, "last_kick_ball_moved_m": 0.183, "kicks": 2, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.48828125, "cy": 0.755859375, "area": 0.033111572265625, "confidence": 1.0, "bearing_deg": 1.3, "est_distance_m": 0.244}], "last_result": {"verb": "walk_to", "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +1°", "data": {"distance_m": 0.244, "bearing_deg": 1.3, "ticks": 12}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 25.397, "kind": "llm", "step": 6, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "826271284", "name": "kick", "arguments": {"leg": "right"}}], "usage": {"input_tokens": 3570, "output_tokens": 25}, "stop_reason": "tool_calls"} +{"t": 25.406, "kind": "verb", "step": 7, "name": "kick", "canonical": "kick", "params": {"leg": "right"}, "ok": true, "summary": "kicked with right leg, ball moved 0.44 m", "data": {"ball_moved_m": 0.444}} +{"t": 25.407, "kind": "frame", "path": "frames/0077.png", "caption": "step 7: kick"} +{"t": 25.409, "kind": "observation", "step": 7, "text": "[step 7/40 · step 7/40, llm calls 7/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.26, 0.69, 2.60 rad)\ncamera: ball at bearing 20° left ~0.66 m\nlast verb `kick`: ok — kicked with right leg, ball moved 0.44 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 13.800000000000061, "x": -0.2585395167483543, "y": 0.688311967025222, "theta": 2.598946723755427, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.724, "holding": false, "extras": {"sim_time": 13.8, "ball": {"x": -0.914, "y": 0.822, "present": true}, "ball_displacement_m": 0.732, "last_kick_ball_moved_m": 0.444, "kicks": 3, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.322265625, "cy": 0.5625, "area": 0.00457763671875, "confidence": 1.0, "bearing_deg": 19.6, "est_distance_m": 0.655}], "last_result": {"verb": "kick", "ok": true, "summary": "kicked with right leg, ball moved 0.44 m", "data": {"ball_moved_m": 0.444}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 28.342, "kind": "llm", "step": 7, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "515048734", "name": "get_frame", "arguments": {}}], "usage": {"input_tokens": 3721, "output_tokens": 23}, "stop_reason": "tool_calls"} +{"t": 28.344, "kind": "frame", "path": "frames/0078.png", "caption": "observe"} +{"t": 28.348, "kind": "verb", "step": 8, "name": "get_frame", "canonical": "observe", "params": {}, "ok": true, "summary": "frame captured; ball at bearing 20° left ~0.66 m", "data": {"detections": [{"label": "ball", "cx": 0.322265625, "cy": 0.5625, "area": 0.00457763671875, "confidence": 1.0, "bearing_deg": 19.6, "est_distance_m": 0.655}]}} +{"t": 28.35, "kind": "frame", "path": "frames/0079.png", "caption": "step 8: get_frame"} +{"t": 28.351, "kind": "observation", "step": 8, "text": "[step 8/40 · step 8/40, llm calls 8/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.26, 0.69, 2.60 rad)\ncamera: ball at bearing 20° left ~0.66 m\nlast verb `get_frame`: ok — frame captured; ball at bearing 20° left ~0.66 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 13.800000000000061, "x": -0.2585395167483543, "y": 0.688311967025222, "theta": 2.598946723755427, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.724, "holding": false, "extras": {"sim_time": 13.8, "ball": {"x": -0.914, "y": 0.822, "present": true}, "ball_displacement_m": 0.732, "last_kick_ball_moved_m": 0.444, "kicks": 3, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.322265625, "cy": 0.5625, "area": 0.00457763671875, "confidence": 1.0, "bearing_deg": 19.6, "est_distance_m": 0.655}], "last_result": {"verb": "get_frame", "ok": true, "summary": "frame captured; ball at bearing 20° left ~0.66 m", "data": {"detections": [{"label": "ball", "cx": 0.322265625, "cy": 0.5625, "area": 0.00457763671875, "confidence": 1.0, "bearing_deg": 19.6, "est_distance_m": 0.655}]}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 32.762, "kind": "llm", "step": 8, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "573502212", "name": "declare_success", "arguments": {"reason": "Ball moved 0.44 m after two kicks, meeting the success criteria."}}], "usage": {"input_tokens": 3875, "output_tokens": 42}, "stop_reason": "tool_calls"} +{"t": 32.762, "kind": "declare", "step": 8, "outcome": "success", "reason": "Ball moved 0.44 m after two kicks, meeting the success criteria."} +{"t": 32.763, "kind": "run_end", "duck": "find-and-kick", "outcome": "success", "reason": "Ball moved 0.44 m after two kicks, meeting the success criteria.", "steps": 8, "llm_calls": 9, "elapsed_s": 13.8, "usage": {"input_tokens": 29403, "output_tokens": 244}, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "transport": "sim2d", "robot": "microduck", "dry_run": false, "final_state": {"t": 13.800000000000061, "x": -0.2585395167483543, "y": 0.688311967025222, "theta": 2.598946723755427, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.724, "holding": false, "extras": {"sim_time": 13.8, "ball": {"x": -0.914, "y": 0.822, "present": true}, "ball_displacement_m": 0.732, "last_kick_ball_moved_m": 0.444, "kicks": 3, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}} diff --git a/docs/local-llms.md b/docs/local-llms.md index 52d9b59..f6217ec 100644 --- a/docs/local-llms.md +++ b/docs/local-llms.md @@ -94,15 +94,22 @@ servers reject image parts. The text observation already carries what the camera ## Honest notes -- Which local model pilots the duck well is an open question we have not measured. The +- Which local model pilots the duck well is an open question we have barely measured. The loop was designed so that a weak planner degrades the task, never the robot's balance. - There is exactly one data point, and it is not ours: the contributor who built memory - between runs ran `find-and-kick` against **Qwen 2.5 Coder 14B on LM Studio**, seeds 5 and - 6, both successes with memory read and written, and reported that the model ignored a - memory hint sitting in the system prompt but followed the same instruction once it was a - numbered step in the `.duck` body. That is one model, one machine, two seeds, and no - transcript in this repository. If you run one, please share the transcript in a - Discussion: it is the cheapest way to make this section shorter. + There are exactly two data points, and they are not ours: the contributor who built + memory between runs ran `find-and-kick` against **Qwen 2.5 Coder 14B on LM Studio** + (Apple M2 Pro, 2026-09-03) and the transcripts are in [`assets/transcripts/`](assets/transcripts/): + + | transcript | seed | outcome | steps | LLM calls | tokens in + out | text fallbacks | what it shows | + |---|---|---|---|---|---|---|---| + | [`…seed6-memory-read.jsonl`](assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl) | 6 | success | 8 | 9 | 29,403 + 244 | 0 | the system prompt carries the previous run's episode under *What you remember*; the model never calls `remember`; two kicks fall short before the third connects | + | [`…seed5-remember.jsonl`](assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl) | 5 | success | 4 | 6 | 17,939 + 265 | 0 | the `.duck` body now says `remember` in strategy step 5; the model calls it with a fact from the verb results, then bundles `quack` and `declare_success` in the same turn (the loop keeps the first, marks `multiple_tool_calls`) | + + Every turn was a native tool call, none needed the JSON text fallback, and the whole + runs took 14 s and 11 s of wall clock. What they cannot show: anything about another + model, another machine, or a harder task than the starter duck. If you run one, please + share the transcript in a Discussion or a PR into that folder: it is the cheapest way + to make this section shorter. - The cloud providers keep their stricter settings (`tool_choice="required"`, `parallel_tool_calls=False`). Only the local presets use the relaxed ones. - Ollama, vLLM, llama.cpp and LM Studio evolve quickly. If a flag above is stale, open an From 81a5a4207399e9ebf85a978340424ec10fc3f8f6 Mon Sep 17 00:00:00 2001 From: Bayway Date: Sun, 6 Sep 2026 10:31:53 +0000 Subject: [PATCH 2/2] docs(transcripts): say what the transcripts say, not more The local-llms table called 14 s and 11 s "wall clock"; that is the simulator clock the budget counts on sim2d. The transcript timestamps give 33 s and 29 s, three to nine seconds per LLM call, and the table now says both. The seed 5 row read as if `remember` came alone and the bundle followed; the model returned all three tools in one response and the loop kept the first. The CHANGELOG entry no longer sits between the Unreleased intro and the audit it introduces. assets/README says why `memory.path` is relative, and memory.md points its "read but never wrote" claim at the two files that show it. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GrWzSyBBfB33MFGYR8qUf9 --- CHANGELOG.md | 14 +++++++------- docs/assets/README.md | 2 +- docs/local-llms.md | 10 ++++++---- docs/memory.md | 2 +- 4 files changed, 15 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b41f3c7..f5ad719 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,13 +7,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -### Local model evidence - -- Two transcripts of `find-and-kick` piloted by **Qwen 2.5 Coder 14B on LM Studio**, seeds - 5 and 6, land in `docs/assets/transcripts/` with a table reading them in - `docs/local-llms.md`, from the contributor whose memory feature they were recorded for. - The README and `local-llms.md` no longer say "no transcript in this repository". - Two hardware paths, each audited against upstream rather than against itself. Still nothing has run on a robot; what changed is that several things which could not have worked now can, and several claims that were not true no longer are. @@ -139,6 +132,13 @@ that is wrong fails as a confident "not fallen". - CI runs on Windows. `robotd` speaks over a unix socket, Windows cannot open one, and the test covering quackd's `ssh -L` answer only runs there — so it had never run anywhere. +### Local model evidence + +- Two transcripts of `find-and-kick` piloted by **Qwen 2.5 Coder 14B on LM Studio**, seeds + 5 and 6, land in `docs/assets/transcripts/` with a table reading them in + `docs/local-llms.md`, from the contributor whose memory feature they were recorded for. + The README and `local-llms.md` no longer say "no transcript in this repository". + ## [0.6.0] — 2026-09-04 A run stops starting from nothing. Every release so far built a pilot with no past: the diff --git a/docs/assets/README.md b/docs/assets/README.md index a86d4c0..c592a51 100644 --- a/docs/assets/README.md +++ b/docs/assets/README.md @@ -9,7 +9,7 @@ | `flock.gif` | Flock mode: three ducks split the search, auction the kick, the closest one takes it. Left: the shared world. Right: the claimant's camera. | `quackd record flock-kick --provider fake --seed 3` — the **scripted planner and deterministic coordinator**, no LLM in the loop (docs/flock.md). | | `hetero.gif` | The 0.4 headline demo: a Reachy Mini head (the slate square on the wall) spots the ball and judges the kick from its own camera while a Microduck walks in and kicks it, two bodies under one contract. Left: the shared world. Right: the acting robot's camera. | `quackd run reachy-spots-duck-kicks --provider fake --seed 3 --gif-size 320` — **scripted pilots and a deterministic coordinator**, zero planner LLM calls. That run ended `success`: the spotter judged the ball moved 0.51 m and the simulator's ground truth agreed. `quackd record` is pinned to `microduck:sim2d`, so this one comes from `run`, which also writes `run.gif`. | | `open-duck.gif` | The 0.5 headline body: an Open Duck Mini v2 finds the ball and walks up to it, because it has no kick. Left: the world from above. Right: the duck's camera. | `quackd run open-duck-scout --robot open_duck:sim2d --provider fake --seed 3 --gif-size 320` — the **scripted pilot**, not an LLM, like every other asset here. | -| `transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl` | A real local model piloting `find-and-kick`: Qwen 2.5 Coder 14B through LM Studio, seed 6, success in 8 steps, with the previous run's episode in the prompt and no `remember` call. | `quackd run find-and-kick --provider lmstudio --model qwen/qwen2.5-coder-14b --seed 6`, copied from `runs//transcript.jsonl` with the contributor's home directory removed from `duck_path`. Read in [docs/local-llms.md](../local-llms.md). | +| `transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl` | A real local model piloting `find-and-kick`: Qwen 2.5 Coder 14B through LM Studio, seed 6, success in 8 steps, with the previous run's episode in the prompt and no `remember` call. | `quackd run find-and-kick --provider lmstudio --model qwen/qwen2.5-coder-14b --seed 6`, copied from `runs//transcript.jsonl` with the contributor's home directory removed from `duck_path`; `memory.path` is relative because the runs used `--memory-dir` next to the checkout, not `~/.quackd`. Read in [docs/local-llms.md](../local-llms.md). | | `transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl` | The same model and duck once `remember` sat in strategy step 5: seed 5, success in 4 steps, one `remember` call with a fact from the verb results. | Same command with `--seed 5`, same scrubbing. | | `contributors.svg` | The circle of faces under *Contributing* in the README: everyone who has contributed, humans only, ordered by lines added, every avatar clipped to the same circle. | Generated by [`contributors.py`](contributors.py) from the GitHub contributors API, avatars inlined so the file needs no network. Regenerated by `.github/workflows/contributors.yml` on every push to `main` and weekly, committed only when it changes. Two hosted services were tried first: one was three days stale, the other could not exclude bots or normalise a non square avatar. | | *(missing)* `reachy-spotter.gif` | **Not recorded yet.** A head with no legs finding the ball with its gaze, the simplest illustration that the verb list follows the body. | Would be `quackd run reachy-spotter --provider fake --seed 3`. | diff --git a/docs/local-llms.md b/docs/local-llms.md index f6217ec..5eb96d1 100644 --- a/docs/local-llms.md +++ b/docs/local-llms.md @@ -103,11 +103,13 @@ servers reject image parts. The text observation already carries what the camera | transcript | seed | outcome | steps | LLM calls | tokens in + out | text fallbacks | what it shows | |---|---|---|---|---|---|---|---| | [`…seed6-memory-read.jsonl`](assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl) | 6 | success | 8 | 9 | 29,403 + 244 | 0 | the system prompt carries the previous run's episode under *What you remember*; the model never calls `remember`; two kicks fall short before the third connects | - | [`…seed5-remember.jsonl`](assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl) | 5 | success | 4 | 6 | 17,939 + 265 | 0 | the `.duck` body now says `remember` in strategy step 5; the model calls it with a fact from the verb results, then bundles `quack` and `declare_success` in the same turn (the loop keeps the first, marks `multiple_tool_calls`) | + | [`…seed5-remember.jsonl`](assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl) | 5 | success | 4 | 6 | 17,939 + 265 | 0 | the `.duck` body now says `remember` in strategy step 5; after the kick the model returns `remember`, `quack` and `declare_success` in one response, the loop keeps the first (a fact from the verb results) and marks `multiple_tool_calls`, and the other two arrive one per turn after | - Every turn was a native tool call, none needed the JSON text fallback, and the whole - runs took 14 s and 11 s of wall clock. What they cannot show: anything about another - model, another machine, or a harder task than the starter duck. If you run one, please + Every turn was a native tool call, none needed the JSON text fallback. The simulator + clock (`elapsed_s` in `run_end`, which is what the budget counts on `sim2d`) says 14 s and + 11 s; the transcript timestamps say 33 s and 29 s of wall clock, three to nine seconds per + LLM call. What they cannot show: anything about another model, another machine, or a + harder task than the starter duck. If you run one, please share the transcript in a Discussion or a PR into that folder: it is the cheapest way to make this section shorter. - The cloud providers keep their stricter settings (`tool_choice="required"`, diff --git a/docs/memory.md b/docs/memory.md index fb25544..c988709 100644 --- a/docs/memory.md +++ b/docs/memory.md @@ -59,7 +59,7 @@ shows every fact it saved. Telling the model about `remember` in the prompt is not enough for a small local model: in the runs this feature's contributor did, Qwen 2.5 Coder 14B read the memory block and never -wrote to it. What works is +wrote to it (both cases are in [`assets/transcripts/`](assets/transcripts/)). What works is putting the call **inside the numbered strategy** of the `.duck` body, right before the declaration (`5. When the ball has moved ≥ 0.3 m, \`remember\` where you found the ball, \`quack\` once and declare success.`), plus a short *Memory* section saying what is worth