diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e0832a..1c35b17 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -851,6 +851,13 @@ that were not true no longer are. HTTP snapshot alternative that needs none. Nobody who is not pointing quackd at a real Microduck needs it, and the H.264 and the ICE inside it have never met a duck. +### Local model evidence + +- Two transcripts of `find-and-kick` piloted by **Qwen 2.5 Coder 14B on LM Studio**, seeds + 5 and 6, land in `docs/assets/transcripts/` with a table reading them in + `docs/local-llms.md`, from the contributor whose memory feature they were recorded for. + The README and `local-llms.md` no longer say "no transcript in this repository". + ## [0.6.0] — 2026-09-04 A run stops starting from nothing. Every release so far built a pilot with no past: the diff --git a/README.md b/README.md index 32469c8..3f246a8 100644 --- a/README.md +++ b/README.md @@ -204,7 +204,7 @@ Version 0.7, simulator and mocks. What has been built, and how far each piece ha | MCP server (`quackd serve-mcp`) | ✅ Claude Code and Claude Desktop, fleets with `--robots` (eight `robot_*` tools, tested in process against the simulator and the mocks), no Claude Desktop session on record | | Memory between runs (`quackd memory`, `remember`) | ✅ one JSONL file per `adapter:backend`, notes and run outcomes into the next prompt, tested end to end offline, 🧪 the `remember` tool itself exercised by one local model on one machine and by no cloud model ([docs/memory.md](docs/memory.md)) | | Providers: anthropic, openai, gemini, grok, fake | ✅ implemented, tested offline, real model hero recording pending an API key | -| Local models (Ollama, vLLM, llama.cpp, LM Studio, any OpenAI compatible server) | ✅ implemented and tested against the OpenAI wire format, 🧪 one live run by a contributor (Qwen 2.5 Coder 14B on LM Studio, two seeds), no transcript in the repo, more welcome | +| Local models (Ollama, vLLM, llama.cpp, LM Studio, any OpenAI compatible server) | ✅ implemented and tested against the OpenAI wire format, 🧪 two live runs by a contributor (Qwen 2.5 Coder 14B on LM Studio, seeds 5 and 6), never on this machine, transcripts in [`docs/assets/transcripts/`](docs/assets/transcripts/), more welcome | | Flock mode (multiple cooperating robots, sim2d) | ✅ deterministic auction and bus, one planner LLM call at most, ground truth checked in tests, 🧪 experimental and simulator only | | Heterogeneous flock (a Reachy Mini head and a Microduck, sim2d) | ✅ `reachy-spots-duck-kicks` 10 of 10 seeds, capability aware auction, the spotter judges from its own frames, ground truth vetoes, 🧪 simulator only | | LAN discovery (`quackd discover`, `quackd announce`, `quackd[lan]`) | ✅ record format and both commands on fakes in the suite, 🧪 real zeroconf exercised once on one machine, never between two ([docs/lan.md](docs/lan.md)) | @@ -524,7 +524,7 @@ The physics simulator costs what physics costs. Measured here on one Windows lap - `grab` is open loop upstream and unreliable here on purpose. `fetch` says so in its file. - A manifest can be smaller than the robot. The LeRobot arm's `real` backend claims no camera and no `pick` until it connects, and even then `pick` appears only when a policy object was injected in code. A rosbridge base over `ws` has no camera verbs unless the address names an image topic, and a ToddlerBot has no `move` unless a walk checkpoint is staged. - Default model IDs for OpenAI, Gemini and Grok were not verified at release. -- Local model quality is unmeasured. The JSON text fallback and the one retry exist because small models often miss native tool calls. One contributor reports `find-and-kick` succeeding against Qwen 2.5 Coder 14B through LM Studio on two seeds, and no transcript from it is in this repository. +- Local model quality is unmeasured. The JSON text fallback and the one retry exist because small models often miss native tool calls. One contributor ran `find-and-kick` against Qwen 2.5 Coder 14B through LM Studio on two seeds, both successes, one of them reading the previous run's memory; the two transcripts are in [`docs/assets/transcripts/`](docs/assets/transcripts/) and read in [docs/local-llms.md](docs/local-llms.md). - Flock mode is simulator only, ships two choreographies and exactly two roles (spotter and kicker), and knows only the Microduck and the Reachy Mini, so an Open Duck cannot join one yet. Separation uses sim ground truth, and two robots share no frame of reference on hardware. - LAN discovery and the MQTT bus have each been exercised once, on one machine. Nothing has crossed to a second machine, the MQTT bus is a library with no `--bus` flag, and a flock across machines also needs a clock across machines, which does not exist yet. diff --git a/docs/assets/README.md b/docs/assets/README.md index 4b38bad..b1d469b 100644 --- a/docs/assets/README.md +++ b/docs/assets/README.md @@ -10,6 +10,8 @@ | `flock.gif` | Flock mode: three ducks split the search, auction the kick, the closest one takes it. Left: the shared world. Right: the claimant's camera. | `quackd record flock-kick --provider fake --seed 3` — the **scripted planner and deterministic coordinator**, no LLM in the loop (docs/flock.md). | | `hetero.gif` | The 0.4 headline demo: a Reachy Mini head (the slate square on the wall) spots the ball and judges the kick from its own camera while a Microduck walks in and kicks it, two bodies under one contract. Left: the shared world. Right: the acting robot's camera. | `quackd run reachy-spots-duck-kicks --provider fake --seed 3 --gif-size 320` — **scripted pilots and a deterministic coordinator**, zero planner LLM calls. That run ended `success`: the spotter judged the ball moved 0.51 m and the simulator's ground truth agreed. `quackd record` is pinned to `microduck:sim2d`, so this one comes from `run`, which also writes `run.gif`. | | `open-duck.gif` | The 0.5 headline body: an Open Duck Mini v2 finds the ball and walks up to it, because it has no kick. Left: the world from above. Right: the duck's camera. | `quackd run open-duck-scout --robot open_duck:sim2d --provider fake --seed 3 --gif-size 320` — the **scripted pilot**, not an LLM, like every other asset here. | +| `transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl` | A real local model piloting `find-and-kick`: Qwen 2.5 Coder 14B through LM Studio, seed 6, success in 8 steps, with the previous run's episode in the prompt and no `remember` call. | `quackd run find-and-kick --provider lmstudio --model qwen/qwen2.5-coder-14b --seed 6`, copied from `runs//transcript.jsonl` with the contributor's home directory removed from `duck_path`; `memory.path` is relative because the runs used `--memory-dir` next to the checkout, not `~/.quackd`. Read in [docs/local-llms.md](../local-llms.md). | +| `transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl` | The same model and duck once `remember` sat in strategy step 5: seed 5, success in 4 steps, one `remember` call with a fact from the verb results. | Same command with `--seed 5`, same scrubbing. | | `contributors.svg` | The circle of faces under *Contributing* in the README: everyone who has contributed, humans only, ordered by lines added, every avatar clipped to the same circle. | Generated by [`contributors.py`](contributors.py) from the GitHub contributors API, avatars inlined so the file needs no network. Regenerated by `.github/workflows/contributors.yml` on every push to `main` and weekly, committed only when it changes. | | *(missing)* `reachy-spotter.gif` | **Not recorded yet.** A head with no legs finding the ball with its gaze, the simplest illustration that the verb list follows the body. | Would be `quackd run reachy-spotter --provider fake --seed 3`. | diff --git a/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl b/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl new file mode 100644 index 0000000..0257b05 --- /dev/null +++ b/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl @@ -0,0 +1,79 @@ +{"t": 0.005, "kind": "run_start", "duck": "find-and-kick", "duck_path": "ducks/find-and-kick.duck", "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "transport": "sim2d", "adapter": "microduck", "robot": {"manifest": 1, "id": "microduck", "vendor": "pollen-robotics", "model": "microduck", "embodiment": "biped", "mobility": "legged", "intents": ["twist", "skill", "gaze", "sound", "pose"], "sensors": ["camera", "battery", "odometry", "imu", "tof"], "verbs": [{"name": "observe", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "report_state", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stop", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "say", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {"text": {"description": "What to say.", "maxLength": 200, "minLength": 1, "title": "Text", "type": "string"}}, "required": ["text"], "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "move", "core": true, "description": "Walk with a velocity for a duration. Use small values; the robot is 25 cm tall.", "params_schema": {"additionalProperties": false, "properties": {"vx": {"default": 0.15, "description": "Forward m/s (negative = back).", "maximum": 0.3, "minimum": -0.3, "title": "Vx", "type": "number"}, "vy": {"default": 0.0, "description": "Left m/s (negative = right).", "maximum": 0.2, "minimum": -0.2, "title": "Vy", "type": "number"}, "wz": {"default": 0.0, "description": "Turn rate rad/s (+ = left).", "maximum": 1.5, "minimum": -1.5, "title": "Wz", "type": "number"}, "duration_s": {"default": 1.0, "description": "How long to hold this velocity.", "exclusiveMinimum": 0, "maximum": 10, "title": "Duration S", "type": "number"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "go_to", "core": true, "description": "Walk toward a detected target and stop at a distance. Closes the loop on the camera itself.", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "description": "Label to approach.", "title": "Target", "type": "string"}, "stop_distance": {"default": 0.25, "description": "Stop this far away (m).", "maximum": 1.5, "minimum": 0.1, "title": "Stop Distance", "type": "number"}, "timeout_s": {"default": 20.0, "exclusiveMinimum": 0, "maximum": 60, "title": "Timeout S", "type": "number"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "search_scan", "core": true, "description": "Rotate in steps, looking for a target. Returns where it was seen.", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "description": "Label to look for (e.g. ball, person).", "title": "Target", "type": "string"}, "step_deg": {"default": 45.0, "description": "Rotation per step.", "maximum": 120, "minimum": 15, "title": "Step Deg", "type": "number"}, "max_steps": {"default": 8, "description": "Steps before giving up (8 x 45° = full turn).", "maximum": 16, "minimum": 1, "title": "Max Steps", "type": "integer"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "approach_and", "core": true, "description": "walk_to a target, then run another verb (kick, grab).", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "title": "Target", "type": "string"}, "stop_distance": {"default": 0.25, "maximum": 1.5, "minimum": 0.1, "title": "Stop Distance", "type": "number"}, "then": {"description": "Verb to run once close (e.g. kick, grab).", "title": "Then", "type": "string"}}, "required": ["then"], "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "sit", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stand", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "kick", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"leg": {"default": "right", "description": "Which leg kicks.", "enum": ["left", "right"], "title": "Leg", "type": "string"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "grab", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stand_up", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "quack", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"text": {"anyOf": [{"maxLength": 200, "type": "string"}, {"type": "null"}], "default": null, "description": "What to say. The robot only has duck sounds; text is mapped to a tone.", "title": "Text"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "gaze", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"direction": {"default": "center", "description": "Where to point the head.", "enum": ["center", "left", "right", "up", "down"], "title": "Direction", "type": "string"}, "bearing_deg": {"anyOf": [{"maximum": 90, "minimum": -90, "type": "number"}, {"type": "null"}], "default": null, "description": "Optional exact bearing (+ = left). Overrides direction.", "title": "Bearing Deg"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}], "preconditions": {"move": ["standing"], "kick": ["standing"], "grab": ["standing"], "sit": ["not_fallen"], "stand": ["not_fallen"], "gaze": ["not_fallen"]}, "safety_authority": {"native": "robotd_deadman", "deadman": true, "heartbeat_hz": 2.0}, "frame": {"reference": "body", "note": ""}, "limits": {"max_vx": 0.3, "max_vy": 0.2, "max_wz": 1.5}, "backend": "sim2d", "blurb": "a small biped duck robot (25 cm, 800 g)", "extras": {}}, "dry_run": false, "contract": {"duck": 0, "name": "find-and-kick", "description": "Search the area for a ball, walk to it, kick it.", "author": "rok", "verbs": {"allow": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"], "confirm": []}, "budgets": {"max_steps": 40, "max_minutes": 5.0, "max_llm_calls": 40}, "success": ["Ball displaced more than 0.3 m in sim, or human confirms the kick landed."], "abort_when": ["Battery below 15%", "Same verb fails 3 times in a row"], "persona": "Determined and cheerful. Quack once when you succeed.", "providers": ["fake", "anthropic", "openai", "gemini", "grok"], "learned_verbs": [], "flock": null, "requires": [], "robots": null}, "system_prompt": "You are the brain of a small biped duck robot (25 cm, 800 g). You are a high-level pilot:\nyou choose ONE verb per turn; the robot's own controllers handle balance and gait, and composite\nverbs like `walk_to` close their own loops on the camera. Do not micro-manage.\n\n## Rules (enforced by the executor — not optional)\n- Call exactly one tool per turn. Never zero, never two.\n- Only these verbs are allowed: search_scan, walk_to, kick, quack, get_frame, stop. Anything else is refused.\n- Budgets: 40 steps, 5 minutes, 40 LLM calls. The run stops when any is hit.\n- Verbs marked confirm (none) ask a human before running.\n- When a success criterion is met, call `declare_success`. If the task is impossible, call `declare_failure`.\n\n## Success criteria\n- Ball displaced more than 0.3 m in sim, or human confirms the kick landed.\n\n## Abort conditions you must respect yourself\n- (none beyond the enforced ones)\n\n## Verbs\n- `search_scan`: Rotate in steps, looking for a target. Returns where it was seen.\n- `walk_to`: Walk toward a detected target and stop at a distance. Closes the loop on the camera itself.\n- `kick`: Kick forward with one leg. Only connects if the ball is < 0.3 m away and roughly ahead.\n- `quack`: Make a duck sound, optionally with text (mapped to a tone).\n- `get_frame`: Capture a camera frame and report what is detected in it.\n- `stop`: Stop moving immediately (zero velocity). Always allowed.\n\n## Persona\nDetermined and cheerful. Quack once when you succeed.\n\n## What you remember from earlier runs on this robot\n(nothing yet — this is the first run on this robot)\n\nCall `remember` (one short sentence) when you learn something worth keeping for next time:\nwhere an object usually is, which strategy worked, what to avoid. It is free: it moves\nnothing and costs no step. Do not save what is already listed above.\n\nYou are in the built-in 2D simulator: a cartoon top-down world. Distances are metres, the arena is about 2 m across, and the ball is orange.\n\n## Task file: find-and-kick — Search the area for a ball, walk to it, kick it.\n\n# Task\n\nYou are piloting a small biped duck robot. Find the ball and kick it.\n\n## Strategy\n\n1. `search_scan` to locate the ball. It rotates in steps and reports detections.\n2. `walk_to` the ball and stop about 0.25 m away. `walk_to` closes the approach loop\n itself using the camera — you do not need to micro-manage it.\n3. `kick`. The result tells you whether the ball moved and by how much.\n4. Verify with a fresh frame or the kick result. If the ball has not moved at least\n 0.3 m, `walk_to` it again and retry the kick.\n5. When the ball has moved ≥ 0.3 m, `remember` where you found the ball, `quack` once\n and declare success.\n\n## Notes\n\n- A kick only connects if the ball is close (< 0.3 m) and roughly in front of you.\n If the kick reports \"missed\", reposition with `walk_to` before kicking again.\n- If `search_scan` finds nothing twice in a row, declare failure rather than looping.\n\n## Memory\n\nBefore you declare success or failure, call `remember` once with one short fact a future\nrun can use: where the ball was when you found it (bearing and rough distance from where\nyou started) or what made the kick connect or miss. Skip it if the prompt already\nremembers the same thing. It costs no step.\n\n## If you cannot call tools natively\nReply with exactly one JSON object and nothing else, in this shape:\n{\"name\": \"\", \"arguments\": {}}\n", "tools": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop", "declare_success", "declare_failure", "remember"], "memory": {"robot": "microduck:sim2d", "path": "../memory-qwen3/microduck-sim2d.jsonl", "notes": 0, "episodes": 0}} +{"t": 0.018, "kind": "frame", "path": "frames/0000.png", "caption": "step 0: start"} +{"t": 0.019, "kind": "observation", "step": 0, "text": "[step 0/40 · step 0/40, llm calls 0/40, 0.0/5 min]\nstate: posture=standing policy=stand battery=100% pose=(0.18, 0.18, 0.10 rad)\ncamera: nothing detected\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 0.0, "x": 0.18300175424722812, "y": 0.18476447384189623, "theta": 0.09629333996427047, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 100.0, "holding": false, "extras": {"sim_time": 0.0, "ball": {"x": -0.321, "y": -0.669, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [], "last_result": null, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 4.492, "kind": "llm", "step": 0, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "902543897", "name": "search_scan", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 2551, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 4.493, "kind": "frame", "path": "frames/0001.png", "caption": "search_scan 0"} +{"t": 4.501, "kind": "frame", "path": "frames/0002.png", "caption": "search_scan 1"} +{"t": 4.512, "kind": "frame", "path": "frames/0003.png", "caption": "search_scan 2"} +{"t": 4.52, "kind": "frame", "path": "frames/0004.png", "caption": "search_scan 3"} +{"t": 4.527, "kind": "frame", "path": "frames/0005.png", "caption": "search_scan 4"} +{"t": 4.535, "kind": "frame", "path": "frames/0006.png", "caption": "search_scan 5"} +{"t": 4.537, "kind": "verb", "step": 1, "name": "search_scan", "canonical": "search_scan", "params": {"target": "ball"}, "ok": true, "summary": "ball found: ball at bearing 5° left ~1.00 m (after 5 turn steps)", "data": {"detections": [{"label": "ball", "cx": 0.453125, "cy": 0.5234375, "area": 0.0019683837890625, "confidence": 0.8225, "bearing_deg": 5.4, "est_distance_m": 0.999}], "steps": 5}} +{"t": 4.539, "kind": "frame", "path": "frames/0007.png", "caption": "step 1: search_scan"} +{"t": 4.542, "kind": "observation", "step": 1, "text": "[step 1/40 · step 1/40, llm calls 1/40, 0.1/5 min]\nstate: posture=standing policy=stand battery=100% pose=(0.18, 0.18, -2.20 rad)\ncamera: ball at bearing 5° left ~1.00 m; person at bearing 9° right ~1.23 m\nlast verb `search_scan`: ok — ball found: ball at bearing 5° left ~1.00 m (after 5 turn steps)\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 4.499999999999992, "x": 0.18300175424722812, "y": 0.18476447384189623, "theta": -2.1962847618103494, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.91, "holding": false, "extras": {"sim_time": 4.5, "ball": {"x": -0.321, "y": -0.669, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.453125, "cy": 0.5234375, "area": 0.0019683837890625, "confidence": 0.8225, "bearing_deg": 5.4, "est_distance_m": 0.999}, {"label": "person", "cx": 0.580078125, "cy": 0.4296875, "area": 0.0179443359375, "confidence": 1.0, "bearing_deg": -9.1, "est_distance_m": 1.229}], "last_result": {"verb": "search_scan", "ok": true, "summary": "ball found: ball at bearing 5° left ~1.00 m (after 5 turn steps)", "data": {"detections": [{"label": "ball", "cx": 0.453125, "cy": 0.5234375, "area": 0.0019683837890625, "confidence": 0.8225, "bearing_deg": 5.4, "est_distance_m": 0.999}], "steps": 5}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 8.376, "kind": "llm", "step": 1, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "929129346", "name": "walk_to", "arguments": {"target": "ball", "stop_distance": 0.25}}], "usage": {"input_tokens": 2726, "output_tokens": 35}, "stop_reason": "tool_calls"} +{"t": 8.378, "kind": "frame", "path": "frames/0008.png", "caption": "go_to ball"} +{"t": 8.383, "kind": "frame", "path": "frames/0009.png", "caption": "go_to ball"} +{"t": 8.386, "kind": "frame", "path": "frames/0010.png", "caption": "go_to ball"} +{"t": 8.389, "kind": "frame", "path": "frames/0011.png", "caption": "go_to ball"} +{"t": 8.391, "kind": "frame", "path": "frames/0012.png", "caption": "go_to ball"} +{"t": 8.393, "kind": "frame", "path": "frames/0013.png", "caption": "go_to ball"} +{"t": 8.396, "kind": "frame", "path": "frames/0014.png", "caption": "go_to ball"} +{"t": 8.398, "kind": "frame", "path": "frames/0015.png", "caption": "go_to ball"} +{"t": 8.401, "kind": "frame", "path": "frames/0016.png", "caption": "go_to ball"} +{"t": 8.403, "kind": "frame", "path": "frames/0017.png", "caption": "go_to ball"} +{"t": 8.406, "kind": "frame", "path": "frames/0018.png", "caption": "go_to ball"} +{"t": 8.408, "kind": "frame", "path": "frames/0019.png", "caption": "go_to ball"} +{"t": 8.41, "kind": "frame", "path": "frames/0020.png", "caption": "go_to ball"} +{"t": 8.412, "kind": "frame", "path": "frames/0021.png", "caption": "go_to ball"} +{"t": 8.414, "kind": "frame", "path": "frames/0022.png", "caption": "go_to ball"} +{"t": 8.417, "kind": "frame", "path": "frames/0023.png", "caption": "go_to ball"} +{"t": 8.419, "kind": "frame", "path": "frames/0024.png", "caption": "go_to ball"} +{"t": 8.421, "kind": "frame", "path": "frames/0025.png", "caption": "go_to ball"} +{"t": 8.423, "kind": "frame", "path": "frames/0026.png", "caption": "go_to ball"} +{"t": 8.426, "kind": "frame", "path": "frames/0027.png", "caption": "go_to ball"} +{"t": 8.428, "kind": "frame", "path": "frames/0028.png", "caption": "go_to ball"} +{"t": 8.431, "kind": "frame", "path": "frames/0029.png", "caption": "go_to ball"} +{"t": 8.433, "kind": "frame", "path": "frames/0030.png", "caption": "go_to ball"} +{"t": 8.435, "kind": "frame", "path": "frames/0031.png", "caption": "go_to ball"} +{"t": 8.438, "kind": "frame", "path": "frames/0032.png", "caption": "go_to ball"} +{"t": 8.44, "kind": "frame", "path": "frames/0033.png", "caption": "go_to ball"} +{"t": 8.442, "kind": "frame", "path": "frames/0034.png", "caption": "go_to ball"} +{"t": 8.445, "kind": "frame", "path": "frames/0035.png", "caption": "go_to ball"} +{"t": 8.447, "kind": "frame", "path": "frames/0036.png", "caption": "go_to ball"} +{"t": 8.449, "kind": "frame", "path": "frames/0037.png", "caption": "go_to ball"} +{"t": 8.451, "kind": "frame", "path": "frames/0038.png", "caption": "go_to ball"} +{"t": 8.453, "kind": "frame", "path": "frames/0039.png", "caption": "go_to ball"} +{"t": 8.456, "kind": "frame", "path": "frames/0040.png", "caption": "go_to ball"} +{"t": 8.458, "kind": "frame", "path": "frames/0041.png", "caption": "go_to ball"} +{"t": 8.46, "kind": "frame", "path": "frames/0042.png", "caption": "go_to ball"} +{"t": 8.462, "kind": "frame", "path": "frames/0043.png", "caption": "go_to ball"} +{"t": 8.465, "kind": "frame", "path": "frames/0044.png", "caption": "go_to ball"} +{"t": 8.467, "kind": "frame", "path": "frames/0045.png", "caption": "go_to ball"} +{"t": 8.469, "kind": "frame", "path": "frames/0046.png", "caption": "go_to ball"} +{"t": 8.472, "kind": "frame", "path": "frames/0047.png", "caption": "go_to ball"} +{"t": 8.474, "kind": "frame", "path": "frames/0048.png", "caption": "go_to ball"} +{"t": 8.476, "kind": "frame", "path": "frames/0049.png", "caption": "go_to ball"} +{"t": 8.479, "kind": "frame", "path": "frames/0050.png", "caption": "go_to ball"} +{"t": 8.481, "kind": "frame", "path": "frames/0051.png", "caption": "go_to ball"} +{"t": 8.483, "kind": "frame", "path": "frames/0052.png", "caption": "go_to ball"} +{"t": 8.485, "kind": "frame", "path": "frames/0053.png", "caption": "go_to ball"} +{"t": 8.487, "kind": "verb", "step": 2, "name": "walk_to", "canonical": "go_to", "params": {"target": "ball", "stop_distance": 0.25}, "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.242, "bearing_deg": 0.0, "ticks": 46}} +{"t": 8.488, "kind": "frame", "path": "frames/0054.png", "caption": "step 2: walk_to"} +{"t": 8.489, "kind": "observation", "step": 2, "text": "[step 2/40 · step 2/40, llm calls 2/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.20, -0.46, -2.10 rad)\ncamera: ball dead ahead ~0.24 m; person at bearing 33° right ~0.58 m\nlast verb `walk_to`: ok — reached the ball: ~0.24 m away, bearing +0°\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 9.099999999999994, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.818, "holding": false, "extras": {"sim_time": 9.1, "ball": {"x": -0.321, "y": -0.669, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.5, "cy": 0.7578125, "area": 0.0335845947265625, "confidence": 1.0, "bearing_deg": -0.0, "est_distance_m": 0.242}, {"label": "person", "cx": 0.830078125, "cy": 0.40625, "area": 0.08648681640625, "confidence": 1.0, "bearing_deg": -33.4, "est_distance_m": 0.58}], "last_result": {"verb": "walk_to", "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.242, "bearing_deg": 0.0, "ticks": 46}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 11.63, "kind": "llm", "step": 2, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "380454315", "name": "kick", "arguments": {"leg": "right"}}], "usage": {"input_tokens": 2900, "output_tokens": 25}, "stop_reason": "tool_calls"} +{"t": 11.643, "kind": "verb", "step": 3, "name": "kick", "canonical": "kick", "params": {"leg": "right"}, "ok": true, "summary": "kicked with right leg, ball moved 0.38 m", "data": {"ball_moved_m": 0.379}} +{"t": 11.645, "kind": "frame", "path": "frames/0055.png", "caption": "step 3: kick"} +{"t": 11.646, "kind": "observation", "step": 3, "text": "[step 3/40 · step 3/40, llm calls 3/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.20, -0.46, -2.10 rad)\ncamera: ball at bearing 19° right ~0.61 m; person at bearing 33° right ~0.58 m\nlast verb `kick`: ok — kicked with right leg, ball moved 0.38 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 10.600000000000016, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.788, "holding": false, "extras": {"sim_time": 10.6, "ball": {"x": -0.652, "y": -0.854, "present": true}, "ball_displacement_m": 0.379, "last_kick_ball_moved_m": 0.379, "kicks": 1, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.66796875, "cy": 0.57421875, "area": 0.0052642822265625, "confidence": 1.0, "bearing_deg": -18.6, "est_distance_m": 0.611}, {"label": "person", "cx": 0.830078125, "cy": 0.40625, "area": 0.08648681640625, "confidence": 1.0, "bearing_deg": -33.4, "est_distance_m": 0.58}], "last_result": {"verb": "kick", "ok": true, "summary": "kicked with right leg, ball moved 0.38 m", "data": {"ball_moved_m": 0.379}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 20.794, "kind": "llm", "step": 3, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": "\n\n\n\n", "tool_calls": [{"id": "888225386", "name": "remember", "arguments": {"text": "the ball was found near the starting position and kicked 0.38 m to the right with the right leg"}}, {"id": "217792846", "name": "quack", "arguments": {}}, {"id": "628057790", "name": "declare_success", "arguments": {"reason": "the ball was successfully kicked 0.38 m to the right with the right leg"}}], "usage": {"input_tokens": 3066, "output_tokens": 111}, "stop_reason": "tool_calls"} +{"t": 20.794, "kind": "enforce", "step": 3, "issue": "multiple_tool_calls", "action": "first_only"} +{"t": 20.797, "kind": "memory", "step": 3, "ok": true, "text": "the ball was found near the starting position and kicked 0.38 m to the right with the right leg", "summary": "remembered for future runs: the ball was found near the starting position and kicked 0.38 m to the right with the right leg"} +{"t": 20.801, "kind": "frame", "path": "frames/0056.png", "caption": "step 3: remember"} +{"t": 20.804, "kind": "observation", "step": 3, "text": "[step 3/40 · step 3/40, llm calls 4/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.20, -0.46, -2.10 rad)\ncamera: ball at bearing 19° right ~0.61 m; person at bearing 33° right ~0.58 m\nlast verb `remember`: ok — remembered for future runs: the ball was found near the starting position and kicked 0.38 m to the right with the right leg\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 10.600000000000016, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.788, "holding": false, "extras": {"sim_time": 10.6, "ball": {"x": -0.652, "y": -0.854, "present": true}, "ball_displacement_m": 0.379, "last_kick_ball_moved_m": 0.379, "kicks": 1, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.66796875, "cy": 0.57421875, "area": 0.0052642822265625, "confidence": 1.0, "bearing_deg": -18.6, "est_distance_m": 0.611}, {"label": "person", "cx": 0.830078125, "cy": 0.40625, "area": 0.08648681640625, "confidence": 1.0, "bearing_deg": -33.4, "est_distance_m": 0.58}], "last_result": {"verb": "remember", "ok": true, "summary": "remembered for future runs: the ball was found near the starting position and kicked 0.38 m to the right with the right leg", "data": {"notes": 1}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 24.265, "kind": "llm", "step": 3, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "216390142", "name": "quack", "arguments": {}}], "usage": {"input_tokens": 3269, "output_tokens": 23}, "stop_reason": "tool_calls"} +{"t": 24.265, "kind": "verb", "step": 4, "name": "quack", "canonical": "quack", "params": {}, "ok": true, "summary": "quacked [chirp]", "data": {"tag": "chirp", "text": null}} +{"t": 24.267, "kind": "frame", "path": "frames/0057.png", "caption": "step 4: quack"} +{"t": 24.268, "kind": "observation", "step": 4, "text": "[step 4/40 · step 4/40, llm calls 5/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.20, -0.46, -2.10 rad)\ncamera: ball at bearing 19° right ~0.61 m; person at bearing 33° right ~0.58 m\nlast verb `quack`: ok — quacked [chirp]\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 10.600000000000016, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.788, "holding": false, "extras": {"sim_time": 10.6, "ball": {"x": -0.652, "y": -0.854, "present": true}, "ball_displacement_m": 0.379, "last_kick_ball_moved_m": 0.379, "kicks": 1, "quacks": 1, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.66796875, "cy": 0.57421875, "area": 0.0052642822265625, "confidence": 1.0, "bearing_deg": -18.6, "est_distance_m": 0.611}, {"label": "person", "cx": 0.830078125, "cy": 0.40625, "area": 0.08648681640625, "confidence": 1.0, "bearing_deg": -33.4, "est_distance_m": 0.58}], "last_result": {"verb": "quack", "ok": true, "summary": "quacked [chirp]", "data": {"tag": "chirp", "text": null}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 28.907, "kind": "llm", "step": 4, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "986592274", "name": "declare_success", "arguments": {"reason": "the ball was kicked successfully, moving 0.38 m to the right with the right leg"}}], "usage": {"input_tokens": 3427, "output_tokens": 45}, "stop_reason": "tool_calls"} +{"t": 28.907, "kind": "declare", "step": 4, "outcome": "success", "reason": "the ball was kicked successfully, moving 0.38 m to the right with the right leg"} +{"t": 28.908, "kind": "run_end", "duck": "find-and-kick", "outcome": "success", "reason": "the ball was kicked successfully, moving 0.38 m to the right with the right leg", "steps": 4, "llm_calls": 6, "elapsed_s": 10.6, "usage": {"input_tokens": 17939, "output_tokens": 265}, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "transport": "sim2d", "robot": "microduck", "dry_run": false, "final_state": {"t": 10.600000000000016, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.788, "holding": false, "extras": {"sim_time": 10.6, "ball": {"x": -0.652, "y": -0.854, "present": true}, "ball_displacement_m": 0.379, "last_kick_ball_moved_m": 0.379, "kicks": 1, "quacks": 1, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}} diff --git a/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl b/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl new file mode 100644 index 0000000..a00c2b7 --- /dev/null +++ b/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl @@ -0,0 +1,109 @@ +{"t": 0.006, "kind": "run_start", "duck": "find-and-kick", "duck_path": "ducks/find-and-kick.duck", "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "transport": "sim2d", "adapter": "microduck", "robot": {"manifest": 1, "id": "microduck", "vendor": "pollen-robotics", "model": "microduck", "embodiment": "biped", "mobility": "legged", "intents": ["twist", "skill", "gaze", "sound", "pose"], "sensors": ["camera", "battery", "odometry", "imu", "tof"], "verbs": [{"name": "observe", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "report_state", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stop", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "say", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {"text": {"description": "What to say.", "maxLength": 200, "minLength": 1, "title": "Text", "type": "string"}}, "required": ["text"], "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "move", "core": true, "description": "Walk with a velocity for a duration. Use small values; the robot is 25 cm tall.", "params_schema": {"additionalProperties": false, "properties": {"vx": {"default": 0.15, "description": "Forward m/s (negative = back).", "maximum": 0.3, "minimum": -0.3, "title": "Vx", "type": "number"}, "vy": {"default": 0.0, "description": "Left m/s (negative = right).", "maximum": 0.2, "minimum": -0.2, "title": "Vy", "type": "number"}, "wz": {"default": 0.0, "description": "Turn rate rad/s (+ = left).", "maximum": 1.5, "minimum": -1.5, "title": "Wz", "type": "number"}, "duration_s": {"default": 1.0, "description": "How long to hold this velocity.", "exclusiveMinimum": 0, "maximum": 10, "title": "Duration S", "type": "number"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "go_to", "core": true, "description": "Walk toward a detected target and stop at a distance. Closes the loop on the camera itself.", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "description": "Label to approach.", "title": "Target", "type": "string"}, "stop_distance": {"default": 0.25, "description": "Stop this far away (m).", "maximum": 1.5, "minimum": 0.1, "title": "Stop Distance", "type": "number"}, "timeout_s": {"default": 20.0, "exclusiveMinimum": 0, "maximum": 60, "title": "Timeout S", "type": "number"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "search_scan", "core": true, "description": "Rotate in steps, looking for a target. Returns where it was seen.", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "description": "Label to look for (e.g. ball, person).", "title": "Target", "type": "string"}, "step_deg": {"default": 45.0, "description": "Rotation per step.", "maximum": 120, "minimum": 15, "title": "Step Deg", "type": "number"}, "max_steps": {"default": 8, "description": "Steps before giving up (8 x 45° = full turn).", "maximum": 16, "minimum": 1, "title": "Max Steps", "type": "integer"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "approach_and", "core": true, "description": "walk_to a target, then run another verb (kick, grab).", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "title": "Target", "type": "string"}, "stop_distance": {"default": 0.25, "maximum": 1.5, "minimum": 0.1, "title": "Stop Distance", "type": "number"}, "then": {"description": "Verb to run once close (e.g. kick, grab).", "title": "Then", "type": "string"}}, "required": ["then"], "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "sit", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stand", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "kick", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"leg": {"default": "right", "description": "Which leg kicks.", "enum": ["left", "right"], "title": "Leg", "type": "string"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "grab", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stand_up", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "quack", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"text": {"anyOf": [{"maxLength": 200, "type": "string"}, {"type": "null"}], "default": null, "description": "What to say. The robot only has duck sounds; text is mapped to a tone.", "title": "Text"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "gaze", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"direction": {"default": "center", "description": "Where to point the head.", "enum": ["center", "left", "right", "up", "down"], "title": "Direction", "type": "string"}, "bearing_deg": {"anyOf": [{"maximum": 90, "minimum": -90, "type": "number"}, {"type": "null"}], "default": null, "description": "Optional exact bearing (+ = left). Overrides direction.", "title": "Bearing Deg"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}], "preconditions": {"move": ["standing"], "kick": ["standing"], "grab": ["standing"], "sit": ["not_fallen"], "stand": ["not_fallen"], "gaze": ["not_fallen"]}, "safety_authority": {"native": "robotd_deadman", "deadman": true, "heartbeat_hz": 2.0}, "frame": {"reference": "body", "note": ""}, "limits": {"max_vx": 0.3, "max_vy": 0.2, "max_wz": 1.5}, "backend": "sim2d", "blurb": "a small biped duck robot (25 cm, 800 g)", "extras": {}}, "dry_run": false, "contract": {"duck": 0, "name": "find-and-kick", "description": "Search the area for a ball, walk to it, kick it.", "author": "rok", "verbs": {"allow": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"], "confirm": []}, "budgets": {"max_steps": 40, "max_minutes": 5.0, "max_llm_calls": 40}, "success": ["Ball displaced more than 0.3 m in sim, or human confirms the kick landed."], "abort_when": ["Battery below 15%", "Same verb fails 3 times in a row"], "persona": "Determined and cheerful. Quack once when you succeed.", "providers": ["fake", "anthropic", "openai", "gemini", "grok"], "learned_verbs": [], "flock": null, "requires": [], "robots": null}, "system_prompt": "You are the brain of a small biped duck robot (25 cm, 800 g). You are a high-level pilot:\nyou choose ONE verb per turn; the robot's own controllers handle balance and gait, and composite\nverbs like `walk_to` close their own loops on the camera. Do not micro-manage.\n\n## Rules (enforced by the executor — not optional)\n- Call exactly one tool per turn. Never zero, never two.\n- Only these verbs are allowed: search_scan, walk_to, kick, quack, get_frame, stop. Anything else is refused.\n- Budgets: 40 steps, 5 minutes, 40 LLM calls. The run stops when any is hit.\n- Verbs marked confirm (none) ask a human before running.\n- When a success criterion is met, call `declare_success`. If the task is impossible, call `declare_failure`.\n\n## Success criteria\n- Ball displaced more than 0.3 m in sim, or human confirms the kick landed.\n\n## Abort conditions you must respect yourself\n- (none beyond the enforced ones)\n\n## Verbs\n- `search_scan`: Rotate in steps, looking for a target. Returns where it was seen.\n- `walk_to`: Walk toward a detected target and stop at a distance. Closes the loop on the camera itself.\n- `kick`: Kick forward with one leg. Only connects if the ball is < 0.3 m away and roughly ahead.\n- `quack`: Make a duck sound, optionally with text (mapped to a tone).\n- `get_frame`: Capture a camera frame and report what is detected in it.\n- `stop`: Stop moving immediately (zero velocity). Always allowed.\n\n## Persona\nDetermined and cheerful. Quack once when you succeed.\n\n## What you remember from earlier runs on this robot\nYour most recent runs on this robot (newest first):\n- [2026-09-03] find-and-kick: success — Ball moved 0.38 m after kick, meeting the success criteria. (4 steps) · search_scan: ball found: ball at bearing 5° left ~1.00 m (after 5 turn steps); walk_to: reached the ball: ~0.24 m away, bearing +0°; kick: kicked with right leg, ball moved 0.38 m; quack: quacked [chirp]\n\nCall `remember` (one short sentence) when you learn something worth keeping for next time:\nwhere an object usually is, which strategy worked, what to avoid. It is free: it moves\nnothing and costs no step. Do not save what is already listed above.\n\nYou are in the built-in 2D simulator: a cartoon top-down world. Distances are metres, the arena is about 2 m across, and the ball is orange.\n\n## Task file: find-and-kick — Search the area for a ball, walk to it, kick it.\n\n# Task\n\nYou are piloting a small biped duck robot. Find the ball and kick it.\n\n## Strategy\n\n1. `search_scan` to locate the ball. It rotates in steps and reports detections.\n2. `walk_to` the ball and stop about 0.25 m away. `walk_to` closes the approach loop\n itself using the camera — you do not need to micro-manage it.\n3. `kick`. The result tells you whether the ball moved and by how much.\n4. Verify with a fresh frame or the kick result. If the ball has not moved at least\n 0.3 m, `walk_to` it again and retry the kick.\n5. When the ball has moved ≥ 0.3 m, `quack` once and declare success.\n\n## Notes\n\n- A kick only connects if the ball is close (< 0.3 m) and roughly in front of you.\n If the kick reports \"missed\", reposition with `walk_to` before kicking again.\n- If `search_scan` finds nothing twice in a row, declare failure rather than looping.\n\n## Memory\n\nBefore you declare success or failure, call `remember` once with one short fact a future\nrun can use: where the ball was when you found it (bearing and rough distance from where\nyou started) or what made the kick connect or miss. Skip it if the prompt already\nremembers the same thing. It costs no step.\n\n## If you cannot call tools natively\nReply with exactly one JSON object and nothing else, in this shape:\n{\"name\": \"\", \"arguments\": {}}\n", "tools": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop", "declare_success", "declare_failure", "remember"], "memory": {"robot": "microduck:sim2d", "path": "../memory-qwen2/microduck-sim2d.jsonl", "notes": 0, "episodes": 1}} +{"t": 0.018, "kind": "frame", "path": "frames/0000.png", "caption": "step 0: start"} +{"t": 0.019, "kind": "observation", "step": 0, "text": "[step 0/40 · step 0/40, llm calls 0/40, 0.0/5 min]\nstate: posture=standing policy=stand battery=100% pose=(0.02, -0.09, -0.82 rad)\ncamera: nothing detected\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 0.0, "x": 0.022898610883165915, "y": -0.09403747811199693, "theta": -0.8226747951461474, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 100.0, "holding": false, "extras": {"sim_time": 0.0, "ball": {"x": -0.188, "y": 0.731, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [], "last_result": null, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 6.924, "kind": "llm", "step": 0, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "708275807", "name": "search_scan", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 2652, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 6.926, "kind": "frame", "path": "frames/0001.png", "caption": "search_scan 0"} +{"t": 6.933, "kind": "frame", "path": "frames/0002.png", "caption": "search_scan 1"} +{"t": 6.939, "kind": "frame", "path": "frames/0003.png", "caption": "search_scan 2"} +{"t": 6.945, "kind": "frame", "path": "frames/0004.png", "caption": "search_scan 3"} +{"t": 6.947, "kind": "verb", "step": 1, "name": "search_scan", "canonical": "search_scan", "params": {"target": "ball"}, "ok": true, "summary": "ball found: ball at bearing 14° left ~0.83 m (after 3 turn steps)", "data": {"detections": [{"label": "ball", "cx": 0.376953125, "cy": 0.5390625, "area": 0.002838134765625, "confidence": 0.9650000000000001, "bearing_deg": 13.8, "est_distance_m": 0.832}], "steps": 3}} +{"t": 6.948, "kind": "frame", "path": "frames/0005.png", "caption": "step 1: search_scan"} +{"t": 6.95, "kind": "observation", "step": 1, "text": "[step 1/40 · step 1/40, llm calls 1/40, 0.0/5 min]\nstate: posture=standing policy=stand battery=100% pose=(0.02, -0.09, 1.58 rad)\ncamera: ball at bearing 14° left ~0.83 m\nlast verb `search_scan`: ok — ball found: ball at bearing 14° left ~0.83 m (after 3 turn steps)\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 2.6999999999999984, "x": 0.022898610883165915, "y": -0.09403747811199693, "theta": 1.5799555297252343, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.946, "holding": false, "extras": {"sim_time": 2.7, "ball": {"x": -0.188, "y": 0.731, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.376953125, "cy": 0.5390625, "area": 0.002838134765625, "confidence": 0.9650000000000001, "bearing_deg": 13.8, "est_distance_m": 0.832}], "last_result": {"verb": "search_scan", "ok": true, "summary": "ball found: ball at bearing 14° left ~0.83 m (after 3 turn steps)", "data": {"detections": [{"label": "ball", "cx": 0.376953125, "cy": 0.5390625, "area": 0.002838134765625, "confidence": 0.9650000000000001, "bearing_deg": 13.8, "est_distance_m": 0.832}], "steps": 3}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 10.167, "kind": "llm", "step": 1, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "421996211", "name": "walk_to", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 2815, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 10.169, "kind": "frame", "path": "frames/0006.png", "caption": "go_to ball"} +{"t": 10.173, "kind": "frame", "path": "frames/0007.png", "caption": "go_to ball"} +{"t": 10.176, "kind": "frame", "path": "frames/0008.png", "caption": "go_to ball"} +{"t": 10.178, "kind": "frame", "path": "frames/0009.png", "caption": "go_to ball"} +{"t": 10.181, "kind": "frame", "path": "frames/0010.png", "caption": "go_to ball"} +{"t": 10.183, "kind": "frame", "path": "frames/0011.png", "caption": "go_to ball"} +{"t": 10.185, "kind": "frame", "path": "frames/0012.png", "caption": "go_to ball"} +{"t": 10.187, "kind": "frame", "path": "frames/0013.png", "caption": "go_to ball"} +{"t": 10.19, "kind": "frame", "path": "frames/0014.png", "caption": "go_to ball"} +{"t": 10.193, "kind": "frame", "path": "frames/0015.png", "caption": "go_to ball"} +{"t": 10.195, "kind": "frame", "path": "frames/0016.png", "caption": "go_to ball"} +{"t": 10.197, "kind": "frame", "path": "frames/0017.png", "caption": "go_to ball"} +{"t": 10.2, "kind": "frame", "path": "frames/0018.png", "caption": "go_to ball"} +{"t": 10.202, "kind": "frame", "path": "frames/0019.png", "caption": "go_to ball"} +{"t": 10.204, "kind": "frame", "path": "frames/0020.png", "caption": "go_to ball"} +{"t": 10.207, "kind": "frame", "path": "frames/0021.png", "caption": "go_to ball"} +{"t": 10.209, "kind": "frame", "path": "frames/0022.png", "caption": "go_to ball"} +{"t": 10.211, "kind": "frame", "path": "frames/0023.png", "caption": "go_to ball"} +{"t": 10.213, "kind": "frame", "path": "frames/0024.png", "caption": "go_to ball"} +{"t": 10.215, "kind": "frame", "path": "frames/0025.png", "caption": "go_to ball"} +{"t": 10.218, "kind": "frame", "path": "frames/0026.png", "caption": "go_to ball"} +{"t": 10.22, "kind": "frame", "path": "frames/0027.png", "caption": "go_to ball"} +{"t": 10.222, "kind": "frame", "path": "frames/0028.png", "caption": "go_to ball"} +{"t": 10.224, "kind": "frame", "path": "frames/0029.png", "caption": "go_to ball"} +{"t": 10.227, "kind": "frame", "path": "frames/0030.png", "caption": "go_to ball"} +{"t": 10.229, "kind": "frame", "path": "frames/0031.png", "caption": "go_to ball"} +{"t": 10.231, "kind": "frame", "path": "frames/0032.png", "caption": "go_to ball"} +{"t": 10.233, "kind": "frame", "path": "frames/0033.png", "caption": "go_to ball"} +{"t": 10.235, "kind": "frame", "path": "frames/0034.png", "caption": "go_to ball"} +{"t": 10.237, "kind": "frame", "path": "frames/0035.png", "caption": "go_to ball"} +{"t": 10.24, "kind": "frame", "path": "frames/0036.png", "caption": "go_to ball"} +{"t": 10.242, "kind": "frame", "path": "frames/0037.png", "caption": "go_to ball"} +{"t": 10.244, "kind": "frame", "path": "frames/0038.png", "caption": "go_to ball"} +{"t": 10.246, "kind": "frame", "path": "frames/0039.png", "caption": "go_to ball"} +{"t": 10.248, "kind": "frame", "path": "frames/0040.png", "caption": "go_to ball"} +{"t": 10.251, "kind": "frame", "path": "frames/0041.png", "caption": "go_to ball"} +{"t": 10.253, "kind": "frame", "path": "frames/0042.png", "caption": "go_to ball"} +{"t": 10.255, "kind": "frame", "path": "frames/0043.png", "caption": "go_to ball"} +{"t": 10.257, "kind": "frame", "path": "frames/0044.png", "caption": "go_to ball"} +{"t": 10.259, "kind": "verb", "step": 2, "name": "walk_to", "canonical": "go_to", "params": {"target": "ball"}, "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.242, "bearing_deg": 0.0, "ticks": 39}} +{"t": 10.26, "kind": "frame", "path": "frames/0045.png", "caption": "step 2: walk_to"} +{"t": 10.261, "kind": "observation", "step": 2, "text": "[step 2/40 · step 2/40, llm calls 2/40, 0.1/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.12, 0.50, 1.84 rad)\ncamera: ball dead ahead ~0.24 m\nlast verb `walk_to`: ok — reached the ball: ~0.24 m away, bearing +0°\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 6.5999999999999845, "x": -0.1242526964459676, "y": 0.49863001147922587, "theta": 1.8404184250668225, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.868, "holding": false, "extras": {"sim_time": 6.6, "ball": {"x": -0.188, "y": 0.731, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.5, "cy": 0.76171875, "area": 0.0335845947265625, "confidence": 1.0, "bearing_deg": -0.0, "est_distance_m": 0.242}], "last_result": {"verb": "walk_to", "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.242, "bearing_deg": 0.0, "ticks": 39}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 13.217, "kind": "llm", "step": 2, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "401399688", "name": "kick", "arguments": {"leg": "right"}}], "usage": {"input_tokens": 2966, "output_tokens": 25}, "stop_reason": "tool_calls"} +{"t": 13.226, "kind": "verb", "step": 3, "name": "kick", "canonical": "kick", "params": {"leg": "right"}, "ok": true, "summary": "kicked with right leg, ball moved 0.15 m", "data": {"ball_moved_m": 0.152}} +{"t": 13.227, "kind": "frame", "path": "frames/0046.png", "caption": "step 3: kick"} +{"t": 13.229, "kind": "observation", "step": 3, "text": "[step 3/40 · step 3/40, llm calls 3/40, 0.1/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.12, 0.50, 1.84 rad)\ncamera: ball at bearing 10° left ~0.39 m\nlast verb `kick`: ok — kicked with right leg, ball moved 0.15 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 8.09999999999998, "x": -0.1242526964459676, "y": 0.49863001147922587, "theta": 1.8404184250668225, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.838, "holding": false, "extras": {"sim_time": 8.1, "ball": {"x": -0.29, "y": 0.844, "present": true}, "ball_displacement_m": 0.152, "last_kick_ball_moved_m": 0.152, "kicks": 1, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.41015625, "cy": 0.64453125, "area": 0.0130767822265625, "confidence": 1.0, "bearing_deg": 10.2, "est_distance_m": 0.387}], "last_result": {"verb": "kick", "ok": true, "summary": "kicked with right leg, ball moved 0.15 m", "data": {"ball_moved_m": 0.152}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 16.244, "kind": "llm", "step": 3, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "283888237", "name": "walk_to", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 3117, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 16.247, "kind": "frame", "path": "frames/0047.png", "caption": "go_to ball"} +{"t": 16.25, "kind": "frame", "path": "frames/0048.png", "caption": "go_to ball"} +{"t": 16.252, "kind": "frame", "path": "frames/0049.png", "caption": "go_to ball"} +{"t": 16.255, "kind": "frame", "path": "frames/0050.png", "caption": "go_to ball"} +{"t": 16.257, "kind": "frame", "path": "frames/0051.png", "caption": "go_to ball"} +{"t": 16.259, "kind": "frame", "path": "frames/0052.png", "caption": "go_to ball"} +{"t": 16.262, "kind": "frame", "path": "frames/0053.png", "caption": "go_to ball"} +{"t": 16.264, "kind": "frame", "path": "frames/0054.png", "caption": "go_to ball"} +{"t": 16.266, "kind": "frame", "path": "frames/0055.png", "caption": "go_to ball"} +{"t": 16.268, "kind": "frame", "path": "frames/0056.png", "caption": "go_to ball"} +{"t": 16.27, "kind": "frame", "path": "frames/0057.png", "caption": "go_to ball"} +{"t": 16.274, "kind": "frame", "path": "frames/0058.png", "caption": "go_to ball"} +{"t": 16.276, "kind": "frame", "path": "frames/0059.png", "caption": "go_to ball"} +{"t": 16.279, "kind": "frame", "path": "frames/0060.png", "caption": "go_to ball"} +{"t": 16.281, "kind": "frame", "path": "frames/0061.png", "caption": "go_to ball"} +{"t": 16.283, "kind": "verb", "step": 4, "name": "walk_to", "canonical": "go_to", "params": {"target": "ball"}, "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.244, "bearing_deg": 0.2, "ticks": 15}} +{"t": 16.284, "kind": "frame", "path": "frames/0062.png", "caption": "step 4: walk_to"} +{"t": 16.285, "kind": "observation", "step": 4, "text": "[step 4/40 · step 4/40, llm calls 4/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.18, 0.63, 2.03 rad)\ncamera: ball dead ahead ~0.24 m\nlast verb `walk_to`: ok — reached the ball: ~0.24 m away, bearing +0°\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 9.600000000000001, "x": -0.18148846702554064, "y": 0.6260237651230226, "theta": 2.0316710037782926, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.808, "holding": false, "extras": {"sim_time": 9.6, "ball": {"x": -0.29, "y": 0.844, "present": true}, "ball_displacement_m": 0.152, "last_kick_ball_moved_m": 0.152, "kicks": 1, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.498046875, "cy": 0.7578125, "area": 0.033111572265625, "confidence": 1.0, "bearing_deg": 0.2, "est_distance_m": 0.244}], "last_result": {"verb": "walk_to", "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.244, "bearing_deg": 0.2, "ticks": 15}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 19.264, "kind": "llm", "step": 4, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "961913944", "name": "kick", "arguments": {"leg": "right"}}], "usage": {"input_tokens": 3268, "output_tokens": 25}, "stop_reason": "tool_calls"} +{"t": 19.276, "kind": "verb", "step": 5, "name": "kick", "canonical": "kick", "params": {"leg": "right"}, "ok": true, "summary": "kicked with right leg, ball moved 0.18 m", "data": {"ball_moved_m": 0.183}} +{"t": 19.277, "kind": "frame", "path": "frames/0063.png", "caption": "step 5: kick"} +{"t": 19.278, "kind": "observation", "step": 5, "text": "[step 5/40 · step 5/40, llm calls 5/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.18, 0.63, 2.03 rad)\ncamera: ball at bearing 31° left ~0.34 m\nlast verb `kick`: ok — kicked with right leg, ball moved 0.18 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 11.100000000000023, "x": -0.18148846702554064, "y": 0.6260237651230226, "theta": 2.0316710037782926, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.778, "holding": false, "extras": {"sim_time": 11.1, "ball": {"x": -0.47, "y": 0.809, "present": true}, "ball_displacement_m": 0.293, "last_kick_ball_moved_m": 0.183, "kicks": 2, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.197265625, "cy": 0.66796875, "area": 0.01690673828125, "confidence": 1.0, "bearing_deg": 31.2, "est_distance_m": 0.341}], "last_result": {"verb": "kick", "ok": true, "summary": "kicked with right leg, ball moved 0.18 m", "data": {"ball_moved_m": 0.183}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 22.361, "kind": "llm", "step": 5, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "646573368", "name": "walk_to", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 3419, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 22.364, "kind": "frame", "path": "frames/0064.png", "caption": "go_to ball"} +{"t": 22.37, "kind": "frame", "path": "frames/0065.png", "caption": "go_to ball"} +{"t": 22.373, "kind": "frame", "path": "frames/0066.png", "caption": "go_to ball"} +{"t": 22.375, "kind": "frame", "path": "frames/0067.png", "caption": "go_to ball"} +{"t": 22.378, "kind": "frame", "path": "frames/0068.png", "caption": "go_to ball"} +{"t": 22.38, "kind": "frame", "path": "frames/0069.png", "caption": "go_to ball"} +{"t": 22.383, "kind": "frame", "path": "frames/0070.png", "caption": "go_to ball"} +{"t": 22.385, "kind": "frame", "path": "frames/0071.png", "caption": "go_to ball"} +{"t": 22.388, "kind": "frame", "path": "frames/0072.png", "caption": "go_to ball"} +{"t": 22.39, "kind": "frame", "path": "frames/0073.png", "caption": "go_to ball"} +{"t": 22.392, "kind": "frame", "path": "frames/0074.png", "caption": "go_to ball"} +{"t": 22.394, "kind": "frame", "path": "frames/0075.png", "caption": "go_to ball"} +{"t": 22.396, "kind": "verb", "step": 6, "name": "walk_to", "canonical": "go_to", "params": {"target": "ball"}, "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +1°", "data": {"distance_m": 0.244, "bearing_deg": 1.3, "ticks": 12}} +{"t": 22.397, "kind": "frame", "path": "frames/0076.png", "caption": "step 6: walk_to"} +{"t": 22.398, "kind": "observation", "step": 6, "text": "[step 6/40 · step 6/40, llm calls 6/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.26, 0.69, 2.60 rad)\ncamera: ball dead ahead ~0.24 m\nlast verb `walk_to`: ok — reached the ball: ~0.24 m away, bearing +1°\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 12.30000000000004, "x": -0.2585395167483543, "y": 0.688311967025222, "theta": 2.598946723755427, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.754, "holding": false, "extras": {"sim_time": 12.3, "ball": {"x": -0.47, "y": 0.809, "present": true}, "ball_displacement_m": 0.293, "last_kick_ball_moved_m": 0.183, "kicks": 2, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.48828125, "cy": 0.755859375, "area": 0.033111572265625, "confidence": 1.0, "bearing_deg": 1.3, "est_distance_m": 0.244}], "last_result": {"verb": "walk_to", "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +1°", "data": {"distance_m": 0.244, "bearing_deg": 1.3, "ticks": 12}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 25.397, "kind": "llm", "step": 6, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "826271284", "name": "kick", "arguments": {"leg": "right"}}], "usage": {"input_tokens": 3570, "output_tokens": 25}, "stop_reason": "tool_calls"} +{"t": 25.406, "kind": "verb", "step": 7, "name": "kick", "canonical": "kick", "params": {"leg": "right"}, "ok": true, "summary": "kicked with right leg, ball moved 0.44 m", "data": {"ball_moved_m": 0.444}} +{"t": 25.407, "kind": "frame", "path": "frames/0077.png", "caption": "step 7: kick"} +{"t": 25.409, "kind": "observation", "step": 7, "text": "[step 7/40 · step 7/40, llm calls 7/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.26, 0.69, 2.60 rad)\ncamera: ball at bearing 20° left ~0.66 m\nlast verb `kick`: ok — kicked with right leg, ball moved 0.44 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 13.800000000000061, "x": -0.2585395167483543, "y": 0.688311967025222, "theta": 2.598946723755427, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.724, "holding": false, "extras": {"sim_time": 13.8, "ball": {"x": -0.914, "y": 0.822, "present": true}, "ball_displacement_m": 0.732, "last_kick_ball_moved_m": 0.444, "kicks": 3, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.322265625, "cy": 0.5625, "area": 0.00457763671875, "confidence": 1.0, "bearing_deg": 19.6, "est_distance_m": 0.655}], "last_result": {"verb": "kick", "ok": true, "summary": "kicked with right leg, ball moved 0.44 m", "data": {"ball_moved_m": 0.444}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 28.342, "kind": "llm", "step": 7, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "515048734", "name": "get_frame", "arguments": {}}], "usage": {"input_tokens": 3721, "output_tokens": 23}, "stop_reason": "tool_calls"} +{"t": 28.344, "kind": "frame", "path": "frames/0078.png", "caption": "observe"} +{"t": 28.348, "kind": "verb", "step": 8, "name": "get_frame", "canonical": "observe", "params": {}, "ok": true, "summary": "frame captured; ball at bearing 20° left ~0.66 m", "data": {"detections": [{"label": "ball", "cx": 0.322265625, "cy": 0.5625, "area": 0.00457763671875, "confidence": 1.0, "bearing_deg": 19.6, "est_distance_m": 0.655}]}} +{"t": 28.35, "kind": "frame", "path": "frames/0079.png", "caption": "step 8: get_frame"} +{"t": 28.351, "kind": "observation", "step": 8, "text": "[step 8/40 · step 8/40, llm calls 8/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.26, 0.69, 2.60 rad)\ncamera: ball at bearing 20° left ~0.66 m\nlast verb `get_frame`: ok — frame captured; ball at bearing 20° left ~0.66 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 13.800000000000061, "x": -0.2585395167483543, "y": 0.688311967025222, "theta": 2.598946723755427, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.724, "holding": false, "extras": {"sim_time": 13.8, "ball": {"x": -0.914, "y": 0.822, "present": true}, "ball_displacement_m": 0.732, "last_kick_ball_moved_m": 0.444, "kicks": 3, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.322265625, "cy": 0.5625, "area": 0.00457763671875, "confidence": 1.0, "bearing_deg": 19.6, "est_distance_m": 0.655}], "last_result": {"verb": "get_frame", "ok": true, "summary": "frame captured; ball at bearing 20° left ~0.66 m", "data": {"detections": [{"label": "ball", "cx": 0.322265625, "cy": 0.5625, "area": 0.00457763671875, "confidence": 1.0, "bearing_deg": 19.6, "est_distance_m": 0.655}]}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 32.762, "kind": "llm", "step": 8, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "573502212", "name": "declare_success", "arguments": {"reason": "Ball moved 0.44 m after two kicks, meeting the success criteria."}}], "usage": {"input_tokens": 3875, "output_tokens": 42}, "stop_reason": "tool_calls"} +{"t": 32.762, "kind": "declare", "step": 8, "outcome": "success", "reason": "Ball moved 0.44 m after two kicks, meeting the success criteria."} +{"t": 32.763, "kind": "run_end", "duck": "find-and-kick", "outcome": "success", "reason": "Ball moved 0.44 m after two kicks, meeting the success criteria.", "steps": 8, "llm_calls": 9, "elapsed_s": 13.8, "usage": {"input_tokens": 29403, "output_tokens": 244}, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "transport": "sim2d", "robot": "microduck", "dry_run": false, "final_state": {"t": 13.800000000000061, "x": -0.2585395167483543, "y": 0.688311967025222, "theta": 2.598946723755427, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.724, "holding": false, "extras": {"sim_time": 13.8, "ball": {"x": -0.914, "y": 0.822, "present": true}, "ball_displacement_m": 0.732, "last_kick_ball_moved_m": 0.444, "kicks": 3, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}} diff --git a/docs/local-llms.md b/docs/local-llms.md index de8a1ec..9ab2c7d 100644 --- a/docs/local-llms.md +++ b/docs/local-llms.md @@ -120,15 +120,24 @@ it. What the page has and has not been run against is in [web/README.md](../web/ ## Honest notes -- Which local model pilots the duck well is an open question we have not measured. The +- Which local model pilots the duck well is an open question we have barely measured. The loop was designed so that a weak planner degrades the task, never the robot's balance. - There is exactly one data point, and it is not ours: the contributor who built memory - between runs ran `find-and-kick` against **Qwen 2.5 Coder 14B on LM Studio**, seeds 5 and - 6, both successes with memory read and written, and reported that the model ignored a - memory hint sitting in the system prompt but followed the same instruction once it was a - numbered step in the `.duck` body. That is one model, one machine, two seeds, and no - transcript in this repository. If you run one, please share the transcript in a - Discussion: it is the cheapest way to make this section shorter. + There are exactly two data points, and they are not ours: the contributor who built + memory between runs ran `find-and-kick` against **Qwen 2.5 Coder 14B on LM Studio** + (Apple M2 Pro, 2026-09-03) and the transcripts are in [`assets/transcripts/`](assets/transcripts/): + + | transcript | seed | outcome | steps | LLM calls | tokens in + out | text fallbacks | what it shows | + |---|---|---|---|---|---|---|---| + | [`…seed6-memory-read.jsonl`](assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl) | 6 | success | 8 | 9 | 29,403 + 244 | 0 | the system prompt carries the previous run's episode under *What you remember*; the model never calls `remember`; two kicks fall short before the third connects | + | [`…seed5-remember.jsonl`](assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl) | 5 | success | 4 | 6 | 17,939 + 265 | 0 | the `.duck` body now says `remember` in strategy step 5; after the kick the model returns `remember`, `quack` and `declare_success` in one response, the loop keeps the first (a fact from the verb results) and marks `multiple_tool_calls`, and the other two arrive one per turn after | + + Every turn was a native tool call, none needed the JSON text fallback. The simulator + clock (`elapsed_s` in `run_end`, which is what the budget counts on `sim2d`) says 14 s and + 11 s; the transcript timestamps say 33 s and 29 s of wall clock, three to nine seconds per + LLM call. What they cannot show: anything about another model, another machine, or a + harder task than the starter duck. If you run one, please + share the transcript in a Discussion or a PR into that folder: it is the cheapest way + to make this section shorter. - The cloud providers keep their stricter settings (`tool_choice="required"`, `parallel_tool_calls=False`). Only the local presets use the relaxed ones. - Ollama, vLLM, llama.cpp and LM Studio evolve quickly. If a flag above is stale, open an diff --git a/docs/memory.md b/docs/memory.md index 9678df1..b0a6678 100644 --- a/docs/memory.md +++ b/docs/memory.md @@ -65,7 +65,7 @@ shows every fact it saved. Telling the model about `remember` in the prompt is not enough for a small local model: in the runs this feature's contributor did, Qwen 2.5 Coder 14B read the memory block and never -wrote to it. What works is +wrote to it (both cases are in [`assets/transcripts/`](assets/transcripts/)). What works is putting the call **inside the numbered strategy** of the `.duck` body, right before the declaration (`5. When the ball has moved ≥ 0.3 m, \`remember\` where you found the ball, \`quack\` once and declare success.`), plus a short *Memory* section saying what is worth