diff --git a/CHANGELOG.md b/CHANGELOG.md index 45e4a69..16bd67d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -210,6 +210,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 through `web/src/catalogue.js`, generated from the Python and grouped by status, with a test that fails when the generated file has drifted from `catalogue.py`. +- Two transcripts of `find-and-kick` piloted by **Qwen 2.5 Coder 14B on LM Studio**, seeds + 5 and 6, land in `docs/assets/transcripts/` with a table reading them in + `docs/local-llms.md`, from the contributor whose memory feature they were recorded for. + The README, `local-llms.md` and PLAN.md no longer say "no transcript in this + repository". The pair is not a chain: they ran against different memory directories, + so one shows the write and the other the read, and neither shows a note surviving + from one run into the next. Thanks to [@Bayway](https://github.com/Bayway) (#7), + whose runs and whose transcripts these are. + ### Changed - **The LeRobot arm adapter stops taking the arm's word for four things it never said.** diff --git a/PLAN.md b/PLAN.md index a18369e..db4552a 100644 --- a/PLAN.md +++ b/PLAN.md @@ -103,10 +103,12 @@ not before. one is assumed equal to the training maximum, so every lateral request is sent at full scale. The assumption is in `GAIT_THRESHOLD`'s note and in the state's `assumptions`, and the fix is the same script that produced the other two. -- 🔨 **A transcript from a live local server** (Ollama, vLLM, llama.cpp). None on the dev - machine. PR #5's contributor reports `find-and-kick` against Qwen 2.5 Coder 14B through LM - Studio on seeds 5 and 6, both successes with memory read and written, but no transcript from - it is in the repository, so the README says exactly that. +- 🔨 **A transcript from a live local server other than LM Studio** (Ollama, vLLM, + llama.cpp). Still none on the dev machine. LM Studio is now covered: #7 landed two + `find-and-kick` transcripts against Qwen 2.5 Coder 14B in `docs/assets/transcripts/`, + read in [docs/local-llms.md](docs/local-llms.md). They are one model on one machine on + one afternoon, and they are not a chain, so what is still open is a second server and a + run where a note written by one run is read by the next. - ⏸ **Exercise `remember` against a cloud model.** The scripted pilot has no script for it, so `--provider fake` writes episodes and never a note. - ⏸ Upload `docs/assets/social-preview.png` under Settings → Social preview. There is no API diff --git a/README.md b/README.md index cdacae2..540d878 100644 --- a/README.md +++ b/README.md @@ -215,7 +215,7 @@ Version 0.8, simulator and mocks. What has been built, and how far each piece ha | MCP server (`quackd serve-mcp`) | ✅ Claude Code and Claude Desktop, one robot or a flock with `--robots` or `--flock NAME` (nine `robot_*` tools, tested in process against the simulator and the mocks), no Claude Desktop session on record | | Memory between runs (`quackd memory`, `remember`) | ✅ one JSONL file per `adapter:backend`, or per registered robot name, notes and run outcomes into the next prompt, tested end to end offline, 🧪 the `remember` tool itself exercised by one local model on one machine and by no cloud model ([docs/memory.md](docs/memory.md)) | | Providers: eleven cloud vendors, fake | ✅ implemented, tested offline against stubbed SDK clients, with one hand curated catalogue of 115 model ids that `--model` is checked against before any call (`quackd list-models`), real model hero recording pending an API key | -| Local models (Ollama, vLLM, llama.cpp, LM Studio, any OpenAI compatible server) | ✅ implemented and tested against the OpenAI wire format, 🧪 one live run by a contributor (Qwen 2.5 Coder 14B on LM Studio, two seeds), no transcript in the repo, more welcome | +| Local models (Ollama, vLLM, llama.cpp, LM Studio, any OpenAI compatible server) | ✅ implemented and tested against the OpenAI wire format, 🧪 two live runs by a contributor (Qwen 2.5 Coder 14B on LM Studio, seeds 5 and 6), never on this machine, transcripts in [`docs/assets/transcripts/`](docs/assets/transcripts/), more welcome | | Registered robots and flocks (`quackd robot`, `quackd flock`) | ✅ both command groups over `~/.quackd/robots.json` and `~/.quackd/flocks.json`, so `--robot NAME` means the same thing in every command that takes a robot and `--flock NAME` in `run` and `serve-mcp`, tested offline, `robot list --probe` answered by the mocks, 🧪 never pointed at hardware ([docs/registry.md](docs/registry.md)) | | Pilot flocks, several robots on one task (`--flock NAME`) | ✅ one whole pilot per body, any backend, same or different bodies, 2 to 8, each with its own executor, allowlist, budget, heartbeat, memory and verdict, dividing the work with `tell` over the bus, 🧪 experimental, exercised on mock and sim2d bodies with the scripted pilot, by no real model and on no hardware | | Coordinator flock, one referee instead (`--flock N`) | ✅ deterministic auction and bus, one planner LLM call at most, ground truth checked in tests, 🧪 experimental and sim2d Microducks only. Its capability aware role auction (spotter/kicker) is unit tested but has no bundled two-robot demo today | @@ -630,7 +630,7 @@ The physics simulator costs what physics costs. Measured here on one Windows lap - `grab` is open loop upstream and unreliable here on purpose. `fetch` says so in its file. - A manifest can be smaller than the robot. The LeRobot arm's `real` backend claims no camera and no `pick` until it connects, and even then `pick` appears only when a policy object was injected in code. A rosbridge base over `ws` has no camera verbs unless the address names an image topic, and a ToddlerBot has no `move` unless a walk checkpoint is staged. - The model catalogue is hand curated. It was read off the eleven vendors' own documentation on 2026-09-12 and it is a snapshot of that day, not a live list. A vendor can retire, rename or add an id between quackd releases, and this build would then refuse an id that is real and offer one that is gone. `quackd list-models` prints exactly what this build knows, which is the only thing `--model` accepts for a cloud vendor. -- Local model quality is unmeasured. The JSON text fallback and the one retry exist because small models often miss native tool calls. One contributor reports `find-and-kick` succeeding against Qwen 2.5 Coder 14B through LM Studio on two seeds, and no transcript from it is in this repository. +- Local model quality is unmeasured. The JSON text fallback and the one retry exist because small models often miss native tool calls. One contributor ran `find-and-kick` against Qwen 2.5 Coder 14B through LM Studio on two seeds, both successes, one of them reading an earlier run's memory. The two transcripts are in [`docs/assets/transcripts/`](docs/assets/transcripts/) and read in [docs/local-llms.md](docs/local-llms.md). - The pilot flock takes any body and has run on `mock` and `sim2d` ones only: N simulated members are N separate worlds with no shared arena, it costs one budget and one model call per member per turn, and `tell` has been exercised by the scripted pilot and by no real model. The coordinator flock is simulator only, ships one choreography and exactly two roles (spotter and kicker, unit tested but with no bundled multi-role starter), and knows only the Microduck. Separation uses sim ground truth, and two robots share no frame of reference on hardware. - LAN discovery and the MQTT bus have each been exercised once, on one machine. Nothing has crossed to a second machine, the MQTT bus is a library with no `--bus` flag, and a coordinator flock across machines also needs a clock across machines, which does not exist yet. A pilot flock needs no such clock and has simply never been tried across two. diff --git a/docs/assets/README.md b/docs/assets/README.md index 8543b98..58852eb 100644 --- a/docs/assets/README.md +++ b/docs/assets/README.md @@ -9,6 +9,8 @@ | `transcript-example.jsonl` | The transcript of that run: system prompt, each observation, each tool call, each verb result, token counts. Predates the trace kinds ([ADR-0029](../adr/0029-tracing.md)), so it carries `llm`, `verb` and `frame` and none of `llm_request`, `verb_start`, `gate`, `intent`, `verb_end` or `note`. Re-record it with the next hero, and move the replay test that reads it (`tests/test_cli_trace.py`) onto a fixture of its own first: it is the only old-shape transcript in the repository, and `quackd trace` has to keep reading one. | Copied from `runs//transcript.jsonl`. | | `flock.gif` | Flock mode: three ducks split the search, auction the kick, the closest one takes it. Left: the shared world. Right: the claimant's camera. | `quackd record flock-kick --provider fake --seed 3` — the **scripted planner and deterministic coordinator**, no LLM in the loop (docs/flock.md). | | `open-duck.gif` | The 0.5 headline body: an Open Duck Mini v2 finds the ball and walks up to it, because it has no kick. Left: the world from above. Right: the duck's camera. | `quackd run open-duck-scout --robot open_duck:sim2d --provider fake --seed 3 --gif-size 320` — the **scripted pilot**, not an LLM, like every other asset here. | +| `transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl` | A real local model piloting `find-and-kick`: Qwen 2.5 Coder 14B through LM Studio, seed 6, success in 8 steps, with an earlier run's episode in the prompt and no `remember` call. | `quackd run find-and-kick --provider lmstudio --model qwen/qwen2.5-coder-14b --seed 6`, copied from `runs//transcript.jsonl` with the contributor's home directory removed from `duck_path`; `memory.path` is relative because the runs used `--memory-dir` next to the checkout, not `~/.quackd`. Read in [docs/local-llms.md](../local-llms.md). | +| `transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl` | The same model and duck once `remember` sat in strategy step 5: seed 5, success in 4 steps, one `remember` call with a fact from the verb results. | Same command with `--seed 5`, same scrubbing. | | `contributors.svg` | The circle of faces under *Contributing* in the README: everyone who has contributed, humans only, ordered by lines added, every avatar clipped to the same circle. | Generated by [`contributors.py`](contributors.py) from the GitHub contributors API, avatars inlined so the file needs no network. Regenerated by `.github/workflows/contributors.yml` on every push to `main` and weekly, committed only when it changes. | Every recording here is driven by the scripted pilot rather than a model, and every caption diff --git a/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl b/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl new file mode 100644 index 0000000..0257b05 --- /dev/null +++ b/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl @@ -0,0 +1,79 @@ +{"t": 0.005, "kind": "run_start", "duck": "find-and-kick", "duck_path": "ducks/find-and-kick.duck", "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "transport": "sim2d", "adapter": "microduck", "robot": {"manifest": 1, "id": "microduck", "vendor": "pollen-robotics", "model": "microduck", "embodiment": "biped", "mobility": "legged", "intents": ["twist", "skill", "gaze", "sound", "pose"], "sensors": ["camera", "battery", "odometry", "imu", "tof"], "verbs": [{"name": "observe", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "report_state", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stop", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "say", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {"text": {"description": "What to say.", "maxLength": 200, "minLength": 1, "title": "Text", "type": "string"}}, "required": ["text"], "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "move", "core": true, "description": "Walk with a velocity for a duration. Use small values; the robot is 25 cm tall.", "params_schema": {"additionalProperties": false, "properties": {"vx": {"default": 0.15, "description": "Forward m/s (negative = back).", "maximum": 0.3, "minimum": -0.3, "title": "Vx", "type": "number"}, "vy": {"default": 0.0, "description": "Left m/s (negative = right).", "maximum": 0.2, "minimum": -0.2, "title": "Vy", "type": "number"}, "wz": {"default": 0.0, "description": "Turn rate rad/s (+ = left).", "maximum": 1.5, "minimum": -1.5, "title": "Wz", "type": "number"}, "duration_s": {"default": 1.0, "description": "How long to hold this velocity.", "exclusiveMinimum": 0, "maximum": 10, "title": "Duration S", "type": "number"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "go_to", "core": true, "description": "Walk toward a detected target and stop at a distance. Closes the loop on the camera itself.", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "description": "Label to approach.", "title": "Target", "type": "string"}, "stop_distance": {"default": 0.25, "description": "Stop this far away (m).", "maximum": 1.5, "minimum": 0.1, "title": "Stop Distance", "type": "number"}, "timeout_s": {"default": 20.0, "exclusiveMinimum": 0, "maximum": 60, "title": "Timeout S", "type": "number"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "search_scan", "core": true, "description": "Rotate in steps, looking for a target. Returns where it was seen.", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "description": "Label to look for (e.g. ball, person).", "title": "Target", "type": "string"}, "step_deg": {"default": 45.0, "description": "Rotation per step.", "maximum": 120, "minimum": 15, "title": "Step Deg", "type": "number"}, "max_steps": {"default": 8, "description": "Steps before giving up (8 x 45° = full turn).", "maximum": 16, "minimum": 1, "title": "Max Steps", "type": "integer"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "approach_and", "core": true, "description": "walk_to a target, then run another verb (kick, grab).", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "title": "Target", "type": "string"}, "stop_distance": {"default": 0.25, "maximum": 1.5, "minimum": 0.1, "title": "Stop Distance", "type": "number"}, "then": {"description": "Verb to run once close (e.g. kick, grab).", "title": "Then", "type": "string"}}, "required": ["then"], "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "sit", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stand", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "kick", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"leg": {"default": "right", "description": "Which leg kicks.", "enum": ["left", "right"], "title": "Leg", "type": "string"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "grab", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stand_up", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "quack", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"text": {"anyOf": [{"maxLength": 200, "type": "string"}, {"type": "null"}], "default": null, "description": "What to say. The robot only has duck sounds; text is mapped to a tone.", "title": "Text"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "gaze", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"direction": {"default": "center", "description": "Where to point the head.", "enum": ["center", "left", "right", "up", "down"], "title": "Direction", "type": "string"}, "bearing_deg": {"anyOf": [{"maximum": 90, "minimum": -90, "type": "number"}, {"type": "null"}], "default": null, "description": "Optional exact bearing (+ = left). Overrides direction.", "title": "Bearing Deg"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}], "preconditions": {"move": ["standing"], "kick": ["standing"], "grab": ["standing"], "sit": ["not_fallen"], "stand": ["not_fallen"], "gaze": ["not_fallen"]}, "safety_authority": {"native": "robotd_deadman", "deadman": true, "heartbeat_hz": 2.0}, "frame": {"reference": "body", "note": ""}, "limits": {"max_vx": 0.3, "max_vy": 0.2, "max_wz": 1.5}, "backend": "sim2d", "blurb": "a small biped duck robot (25 cm, 800 g)", "extras": {}}, "dry_run": false, "contract": {"duck": 0, "name": "find-and-kick", "description": "Search the area for a ball, walk to it, kick it.", "author": "rok", "verbs": {"allow": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"], "confirm": []}, "budgets": {"max_steps": 40, "max_minutes": 5.0, "max_llm_calls": 40}, "success": ["Ball displaced more than 0.3 m in sim, or human confirms the kick landed."], "abort_when": ["Battery below 15%", "Same verb fails 3 times in a row"], "persona": "Determined and cheerful. Quack once when you succeed.", "providers": ["fake", "anthropic", "openai", "gemini", "grok"], "learned_verbs": [], "flock": null, "requires": [], "robots": null}, "system_prompt": "You are the brain of a small biped duck robot (25 cm, 800 g). You are a high-level pilot:\nyou choose ONE verb per turn; the robot's own controllers handle balance and gait, and composite\nverbs like `walk_to` close their own loops on the camera. Do not micro-manage.\n\n## Rules (enforced by the executor — not optional)\n- Call exactly one tool per turn. Never zero, never two.\n- Only these verbs are allowed: search_scan, walk_to, kick, quack, get_frame, stop. Anything else is refused.\n- Budgets: 40 steps, 5 minutes, 40 LLM calls. The run stops when any is hit.\n- Verbs marked confirm (none) ask a human before running.\n- When a success criterion is met, call `declare_success`. If the task is impossible, call `declare_failure`.\n\n## Success criteria\n- Ball displaced more than 0.3 m in sim, or human confirms the kick landed.\n\n## Abort conditions you must respect yourself\n- (none beyond the enforced ones)\n\n## Verbs\n- `search_scan`: Rotate in steps, looking for a target. Returns where it was seen.\n- `walk_to`: Walk toward a detected target and stop at a distance. Closes the loop on the camera itself.\n- `kick`: Kick forward with one leg. Only connects if the ball is < 0.3 m away and roughly ahead.\n- `quack`: Make a duck sound, optionally with text (mapped to a tone).\n- `get_frame`: Capture a camera frame and report what is detected in it.\n- `stop`: Stop moving immediately (zero velocity). Always allowed.\n\n## Persona\nDetermined and cheerful. Quack once when you succeed.\n\n## What you remember from earlier runs on this robot\n(nothing yet — this is the first run on this robot)\n\nCall `remember` (one short sentence) when you learn something worth keeping for next time:\nwhere an object usually is, which strategy worked, what to avoid. It is free: it moves\nnothing and costs no step. Do not save what is already listed above.\n\nYou are in the built-in 2D simulator: a cartoon top-down world. Distances are metres, the arena is about 2 m across, and the ball is orange.\n\n## Task file: find-and-kick — Search the area for a ball, walk to it, kick it.\n\n# Task\n\nYou are piloting a small biped duck robot. Find the ball and kick it.\n\n## Strategy\n\n1. `search_scan` to locate the ball. It rotates in steps and reports detections.\n2. `walk_to` the ball and stop about 0.25 m away. `walk_to` closes the approach loop\n itself using the camera — you do not need to micro-manage it.\n3. `kick`. The result tells you whether the ball moved and by how much.\n4. Verify with a fresh frame or the kick result. If the ball has not moved at least\n 0.3 m, `walk_to` it again and retry the kick.\n5. When the ball has moved ≥ 0.3 m, `remember` where you found the ball, `quack` once\n and declare success.\n\n## Notes\n\n- A kick only connects if the ball is close (< 0.3 m) and roughly in front of you.\n If the kick reports \"missed\", reposition with `walk_to` before kicking again.\n- If `search_scan` finds nothing twice in a row, declare failure rather than looping.\n\n## Memory\n\nBefore you declare success or failure, call `remember` once with one short fact a future\nrun can use: where the ball was when you found it (bearing and rough distance from where\nyou started) or what made the kick connect or miss. Skip it if the prompt already\nremembers the same thing. It costs no step.\n\n## If you cannot call tools natively\nReply with exactly one JSON object and nothing else, in this shape:\n{\"name\": \"\", \"arguments\": {}}\n", "tools": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop", "declare_success", "declare_failure", "remember"], "memory": {"robot": "microduck:sim2d", "path": "../memory-qwen3/microduck-sim2d.jsonl", "notes": 0, "episodes": 0}} +{"t": 0.018, "kind": "frame", "path": "frames/0000.png", "caption": "step 0: start"} +{"t": 0.019, "kind": "observation", "step": 0, "text": "[step 0/40 · step 0/40, llm calls 0/40, 0.0/5 min]\nstate: posture=standing policy=stand battery=100% pose=(0.18, 0.18, 0.10 rad)\ncamera: nothing detected\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 0.0, "x": 0.18300175424722812, "y": 0.18476447384189623, "theta": 0.09629333996427047, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 100.0, "holding": false, "extras": {"sim_time": 0.0, "ball": {"x": -0.321, "y": -0.669, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [], "last_result": null, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 4.492, "kind": "llm", "step": 0, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "902543897", "name": "search_scan", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 2551, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 4.493, "kind": "frame", "path": "frames/0001.png", "caption": "search_scan 0"} +{"t": 4.501, "kind": "frame", "path": "frames/0002.png", "caption": "search_scan 1"} +{"t": 4.512, "kind": "frame", "path": "frames/0003.png", "caption": "search_scan 2"} +{"t": 4.52, "kind": "frame", "path": "frames/0004.png", "caption": "search_scan 3"} +{"t": 4.527, "kind": "frame", "path": "frames/0005.png", "caption": "search_scan 4"} +{"t": 4.535, "kind": "frame", "path": "frames/0006.png", "caption": "search_scan 5"} +{"t": 4.537, "kind": "verb", "step": 1, "name": "search_scan", "canonical": "search_scan", "params": {"target": "ball"}, "ok": true, "summary": "ball found: ball at bearing 5° left ~1.00 m (after 5 turn steps)", "data": {"detections": [{"label": "ball", "cx": 0.453125, "cy": 0.5234375, "area": 0.0019683837890625, "confidence": 0.8225, "bearing_deg": 5.4, "est_distance_m": 0.999}], "steps": 5}} +{"t": 4.539, "kind": "frame", "path": "frames/0007.png", "caption": "step 1: search_scan"} +{"t": 4.542, "kind": "observation", "step": 1, "text": "[step 1/40 · step 1/40, llm calls 1/40, 0.1/5 min]\nstate: posture=standing policy=stand battery=100% pose=(0.18, 0.18, -2.20 rad)\ncamera: ball at bearing 5° left ~1.00 m; person at bearing 9° right ~1.23 m\nlast verb `search_scan`: ok — ball found: ball at bearing 5° left ~1.00 m (after 5 turn steps)\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 4.499999999999992, "x": 0.18300175424722812, "y": 0.18476447384189623, "theta": -2.1962847618103494, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.91, "holding": false, "extras": {"sim_time": 4.5, "ball": {"x": -0.321, "y": -0.669, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.453125, "cy": 0.5234375, "area": 0.0019683837890625, "confidence": 0.8225, "bearing_deg": 5.4, "est_distance_m": 0.999}, {"label": "person", "cx": 0.580078125, "cy": 0.4296875, "area": 0.0179443359375, "confidence": 1.0, "bearing_deg": -9.1, "est_distance_m": 1.229}], "last_result": {"verb": "search_scan", "ok": true, "summary": "ball found: ball at bearing 5° left ~1.00 m (after 5 turn steps)", "data": {"detections": [{"label": "ball", "cx": 0.453125, "cy": 0.5234375, "area": 0.0019683837890625, "confidence": 0.8225, "bearing_deg": 5.4, "est_distance_m": 0.999}], "steps": 5}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 8.376, "kind": "llm", "step": 1, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "929129346", "name": "walk_to", "arguments": {"target": "ball", "stop_distance": 0.25}}], "usage": {"input_tokens": 2726, "output_tokens": 35}, "stop_reason": "tool_calls"} +{"t": 8.378, "kind": "frame", "path": "frames/0008.png", "caption": "go_to ball"} +{"t": 8.383, "kind": "frame", "path": "frames/0009.png", "caption": "go_to ball"} +{"t": 8.386, "kind": "frame", "path": "frames/0010.png", "caption": "go_to ball"} +{"t": 8.389, "kind": "frame", "path": "frames/0011.png", "caption": "go_to ball"} +{"t": 8.391, "kind": "frame", "path": "frames/0012.png", "caption": "go_to ball"} +{"t": 8.393, "kind": "frame", "path": "frames/0013.png", "caption": "go_to ball"} +{"t": 8.396, "kind": "frame", "path": "frames/0014.png", "caption": "go_to ball"} +{"t": 8.398, "kind": "frame", "path": "frames/0015.png", "caption": "go_to ball"} +{"t": 8.401, "kind": "frame", "path": "frames/0016.png", "caption": "go_to ball"} +{"t": 8.403, "kind": "frame", "path": "frames/0017.png", "caption": "go_to ball"} +{"t": 8.406, "kind": "frame", "path": "frames/0018.png", "caption": "go_to ball"} +{"t": 8.408, "kind": "frame", "path": "frames/0019.png", "caption": "go_to ball"} +{"t": 8.41, "kind": "frame", "path": "frames/0020.png", "caption": "go_to ball"} +{"t": 8.412, "kind": "frame", "path": "frames/0021.png", "caption": "go_to ball"} +{"t": 8.414, "kind": "frame", "path": "frames/0022.png", "caption": "go_to ball"} +{"t": 8.417, "kind": "frame", "path": "frames/0023.png", "caption": "go_to ball"} +{"t": 8.419, "kind": "frame", "path": "frames/0024.png", "caption": "go_to ball"} +{"t": 8.421, "kind": "frame", "path": "frames/0025.png", "caption": "go_to ball"} +{"t": 8.423, "kind": "frame", "path": "frames/0026.png", "caption": "go_to ball"} +{"t": 8.426, "kind": "frame", "path": "frames/0027.png", "caption": "go_to ball"} +{"t": 8.428, "kind": "frame", "path": "frames/0028.png", "caption": "go_to ball"} +{"t": 8.431, "kind": "frame", "path": "frames/0029.png", "caption": "go_to ball"} +{"t": 8.433, "kind": "frame", "path": "frames/0030.png", "caption": "go_to ball"} +{"t": 8.435, "kind": "frame", "path": "frames/0031.png", "caption": "go_to ball"} +{"t": 8.438, "kind": "frame", "path": "frames/0032.png", "caption": "go_to ball"} +{"t": 8.44, "kind": "frame", "path": "frames/0033.png", "caption": "go_to ball"} +{"t": 8.442, "kind": "frame", "path": "frames/0034.png", "caption": "go_to ball"} +{"t": 8.445, "kind": "frame", "path": "frames/0035.png", "caption": "go_to ball"} +{"t": 8.447, "kind": "frame", "path": "frames/0036.png", "caption": "go_to ball"} +{"t": 8.449, "kind": "frame", "path": "frames/0037.png", "caption": "go_to ball"} +{"t": 8.451, "kind": "frame", "path": "frames/0038.png", "caption": "go_to ball"} +{"t": 8.453, "kind": "frame", "path": "frames/0039.png", "caption": "go_to ball"} +{"t": 8.456, "kind": "frame", "path": "frames/0040.png", "caption": "go_to ball"} +{"t": 8.458, "kind": "frame", "path": "frames/0041.png", "caption": "go_to ball"} +{"t": 8.46, "kind": "frame", "path": "frames/0042.png", "caption": "go_to ball"} +{"t": 8.462, "kind": "frame", "path": "frames/0043.png", "caption": "go_to ball"} +{"t": 8.465, "kind": "frame", "path": "frames/0044.png", "caption": "go_to ball"} +{"t": 8.467, "kind": "frame", "path": "frames/0045.png", "caption": "go_to ball"} +{"t": 8.469, "kind": "frame", "path": "frames/0046.png", "caption": "go_to ball"} +{"t": 8.472, "kind": "frame", "path": "frames/0047.png", "caption": "go_to ball"} +{"t": 8.474, "kind": "frame", "path": "frames/0048.png", "caption": "go_to ball"} +{"t": 8.476, "kind": "frame", "path": "frames/0049.png", "caption": "go_to ball"} +{"t": 8.479, "kind": "frame", "path": "frames/0050.png", "caption": "go_to ball"} +{"t": 8.481, "kind": "frame", "path": "frames/0051.png", "caption": "go_to ball"} +{"t": 8.483, "kind": "frame", "path": "frames/0052.png", "caption": "go_to ball"} +{"t": 8.485, "kind": "frame", "path": "frames/0053.png", "caption": "go_to ball"} +{"t": 8.487, "kind": "verb", "step": 2, "name": "walk_to", "canonical": "go_to", "params": {"target": "ball", "stop_distance": 0.25}, "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.242, "bearing_deg": 0.0, "ticks": 46}} +{"t": 8.488, "kind": "frame", "path": "frames/0054.png", "caption": "step 2: walk_to"} +{"t": 8.489, "kind": "observation", "step": 2, "text": "[step 2/40 · step 2/40, llm calls 2/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.20, -0.46, -2.10 rad)\ncamera: ball dead ahead ~0.24 m; person at bearing 33° right ~0.58 m\nlast verb `walk_to`: ok — reached the ball: ~0.24 m away, bearing +0°\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 9.099999999999994, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.818, "holding": false, "extras": {"sim_time": 9.1, "ball": {"x": -0.321, "y": -0.669, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.5, "cy": 0.7578125, "area": 0.0335845947265625, "confidence": 1.0, "bearing_deg": -0.0, "est_distance_m": 0.242}, {"label": "person", "cx": 0.830078125, "cy": 0.40625, "area": 0.08648681640625, "confidence": 1.0, "bearing_deg": -33.4, "est_distance_m": 0.58}], "last_result": {"verb": "walk_to", "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.242, "bearing_deg": 0.0, "ticks": 46}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 11.63, "kind": "llm", "step": 2, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "380454315", "name": "kick", "arguments": {"leg": "right"}}], "usage": {"input_tokens": 2900, "output_tokens": 25}, "stop_reason": "tool_calls"} +{"t": 11.643, "kind": "verb", "step": 3, "name": "kick", "canonical": "kick", "params": {"leg": "right"}, "ok": true, "summary": "kicked with right leg, ball moved 0.38 m", "data": {"ball_moved_m": 0.379}} +{"t": 11.645, "kind": "frame", "path": "frames/0055.png", "caption": "step 3: kick"} +{"t": 11.646, "kind": "observation", "step": 3, "text": "[step 3/40 · step 3/40, llm calls 3/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.20, -0.46, -2.10 rad)\ncamera: ball at bearing 19° right ~0.61 m; person at bearing 33° right ~0.58 m\nlast verb `kick`: ok — kicked with right leg, ball moved 0.38 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 10.600000000000016, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.788, "holding": false, "extras": {"sim_time": 10.6, "ball": {"x": -0.652, "y": -0.854, "present": true}, "ball_displacement_m": 0.379, "last_kick_ball_moved_m": 0.379, "kicks": 1, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.66796875, "cy": 0.57421875, "area": 0.0052642822265625, "confidence": 1.0, "bearing_deg": -18.6, "est_distance_m": 0.611}, {"label": "person", "cx": 0.830078125, "cy": 0.40625, "area": 0.08648681640625, "confidence": 1.0, "bearing_deg": -33.4, "est_distance_m": 0.58}], "last_result": {"verb": "kick", "ok": true, "summary": "kicked with right leg, ball moved 0.38 m", "data": {"ball_moved_m": 0.379}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 20.794, "kind": "llm", "step": 3, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": "\n\n\n\n", "tool_calls": [{"id": "888225386", "name": "remember", "arguments": {"text": "the ball was found near the starting position and kicked 0.38 m to the right with the right leg"}}, {"id": "217792846", "name": "quack", "arguments": {}}, {"id": "628057790", "name": "declare_success", "arguments": {"reason": "the ball was successfully kicked 0.38 m to the right with the right leg"}}], "usage": {"input_tokens": 3066, "output_tokens": 111}, "stop_reason": "tool_calls"} +{"t": 20.794, "kind": "enforce", "step": 3, "issue": "multiple_tool_calls", "action": "first_only"} +{"t": 20.797, "kind": "memory", "step": 3, "ok": true, "text": "the ball was found near the starting position and kicked 0.38 m to the right with the right leg", "summary": "remembered for future runs: the ball was found near the starting position and kicked 0.38 m to the right with the right leg"} +{"t": 20.801, "kind": "frame", "path": "frames/0056.png", "caption": "step 3: remember"} +{"t": 20.804, "kind": "observation", "step": 3, "text": "[step 3/40 · step 3/40, llm calls 4/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.20, -0.46, -2.10 rad)\ncamera: ball at bearing 19° right ~0.61 m; person at bearing 33° right ~0.58 m\nlast verb `remember`: ok — remembered for future runs: the ball was found near the starting position and kicked 0.38 m to the right with the right leg\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 10.600000000000016, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.788, "holding": false, "extras": {"sim_time": 10.6, "ball": {"x": -0.652, "y": -0.854, "present": true}, "ball_displacement_m": 0.379, "last_kick_ball_moved_m": 0.379, "kicks": 1, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.66796875, "cy": 0.57421875, "area": 0.0052642822265625, "confidence": 1.0, "bearing_deg": -18.6, "est_distance_m": 0.611}, {"label": "person", "cx": 0.830078125, "cy": 0.40625, "area": 0.08648681640625, "confidence": 1.0, "bearing_deg": -33.4, "est_distance_m": 0.58}], "last_result": {"verb": "remember", "ok": true, "summary": "remembered for future runs: the ball was found near the starting position and kicked 0.38 m to the right with the right leg", "data": {"notes": 1}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 24.265, "kind": "llm", "step": 3, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "216390142", "name": "quack", "arguments": {}}], "usage": {"input_tokens": 3269, "output_tokens": 23}, "stop_reason": "tool_calls"} +{"t": 24.265, "kind": "verb", "step": 4, "name": "quack", "canonical": "quack", "params": {}, "ok": true, "summary": "quacked [chirp]", "data": {"tag": "chirp", "text": null}} +{"t": 24.267, "kind": "frame", "path": "frames/0057.png", "caption": "step 4: quack"} +{"t": 24.268, "kind": "observation", "step": 4, "text": "[step 4/40 · step 4/40, llm calls 5/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.20, -0.46, -2.10 rad)\ncamera: ball at bearing 19° right ~0.61 m; person at bearing 33° right ~0.58 m\nlast verb `quack`: ok — quacked [chirp]\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 10.600000000000016, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.788, "holding": false, "extras": {"sim_time": 10.6, "ball": {"x": -0.652, "y": -0.854, "present": true}, "ball_displacement_m": 0.379, "last_kick_ball_moved_m": 0.379, "kicks": 1, "quacks": 1, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}, "detections": [{"label": "ball", "cx": 0.66796875, "cy": 0.57421875, "area": 0.0052642822265625, "confidence": 1.0, "bearing_deg": -18.6, "est_distance_m": 0.611}, {"label": "person", "cx": 0.830078125, "cy": 0.40625, "area": 0.08648681640625, "confidence": 1.0, "bearing_deg": -33.4, "est_distance_m": 0.58}], "last_result": {"verb": "quack", "ok": true, "summary": "quacked [chirp]", "data": {"tag": "chirp", "text": null}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 28.907, "kind": "llm", "step": 4, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "986592274", "name": "declare_success", "arguments": {"reason": "the ball was kicked successfully, moving 0.38 m to the right with the right leg"}}], "usage": {"input_tokens": 3427, "output_tokens": 45}, "stop_reason": "tool_calls"} +{"t": 28.907, "kind": "declare", "step": 4, "outcome": "success", "reason": "the ball was kicked successfully, moving 0.38 m to the right with the right leg"} +{"t": 28.908, "kind": "run_end", "duck": "find-and-kick", "outcome": "success", "reason": "the ball was kicked successfully, moving 0.38 m to the right with the right leg", "steps": 4, "llm_calls": 6, "elapsed_s": 10.6, "usage": {"input_tokens": 17939, "output_tokens": 265}, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "transport": "sim2d", "robot": "microduck", "dry_run": false, "final_state": {"t": 10.600000000000016, "x": -0.19879524533218074, "y": -0.45883882312056473, "theta": -2.0982905406508205, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.788, "holding": false, "extras": {"sim_time": 10.6, "ball": {"x": -0.652, "y": -0.854, "present": true}, "ball_displacement_m": 0.379, "last_kick_ball_moved_m": 0.379, "kicks": 1, "quacks": 1, "head_yaw_deg": 0.0, "people": [{"x": -0.728, "y": -0.722}]}}} diff --git a/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl b/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl new file mode 100644 index 0000000..a00c2b7 --- /dev/null +++ b/docs/assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl @@ -0,0 +1,109 @@ +{"t": 0.006, "kind": "run_start", "duck": "find-and-kick", "duck_path": "ducks/find-and-kick.duck", "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "transport": "sim2d", "adapter": "microduck", "robot": {"manifest": 1, "id": "microduck", "vendor": "pollen-robotics", "model": "microduck", "embodiment": "biped", "mobility": "legged", "intents": ["twist", "skill", "gaze", "sound", "pose"], "sensors": ["camera", "battery", "odometry", "imu", "tof"], "verbs": [{"name": "observe", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "report_state", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stop", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "say", "core": true, "description": "", "params_schema": {"additionalProperties": false, "properties": {"text": {"description": "What to say.", "maxLength": 200, "minLength": 1, "title": "Text", "type": "string"}}, "required": ["text"], "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "move", "core": true, "description": "Walk with a velocity for a duration. Use small values; the robot is 25 cm tall.", "params_schema": {"additionalProperties": false, "properties": {"vx": {"default": 0.15, "description": "Forward m/s (negative = back).", "maximum": 0.3, "minimum": -0.3, "title": "Vx", "type": "number"}, "vy": {"default": 0.0, "description": "Left m/s (negative = right).", "maximum": 0.2, "minimum": -0.2, "title": "Vy", "type": "number"}, "wz": {"default": 0.0, "description": "Turn rate rad/s (+ = left).", "maximum": 1.5, "minimum": -1.5, "title": "Wz", "type": "number"}, "duration_s": {"default": 1.0, "description": "How long to hold this velocity.", "exclusiveMinimum": 0, "maximum": 10, "title": "Duration S", "type": "number"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "go_to", "core": true, "description": "Walk toward a detected target and stop at a distance. Closes the loop on the camera itself.", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "description": "Label to approach.", "title": "Target", "type": "string"}, "stop_distance": {"default": 0.25, "description": "Stop this far away (m).", "maximum": 1.5, "minimum": 0.1, "title": "Stop Distance", "type": "number"}, "timeout_s": {"default": 20.0, "exclusiveMinimum": 0, "maximum": 60, "title": "Timeout S", "type": "number"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "search_scan", "core": true, "description": "Rotate in steps, looking for a target. Returns where it was seen.", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "description": "Label to look for (e.g. ball, person).", "title": "Target", "type": "string"}, "step_deg": {"default": 45.0, "description": "Rotation per step.", "maximum": 120, "minimum": 15, "title": "Step Deg", "type": "number"}, "max_steps": {"default": 8, "description": "Steps before giving up (8 x 45° = full turn).", "maximum": 16, "minimum": 1, "title": "Max Steps", "type": "integer"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "approach_and", "core": true, "description": "walk_to a target, then run another verb (kick, grab).", "params_schema": {"additionalProperties": false, "properties": {"target": {"default": "ball", "title": "Target", "type": "string"}, "stop_distance": {"default": 0.25, "maximum": 1.5, "minimum": 0.1, "title": "Stop Distance", "type": "number"}, "then": {"description": "Verb to run once close (e.g. kick, grab).", "title": "Then", "type": "string"}}, "required": ["then"], "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "sit", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stand", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "kick", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"leg": {"default": "right", "description": "Which leg kicks.", "enum": ["left", "right"], "title": "Leg", "type": "string"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "grab", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "stand_up", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "quack", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"text": {"anyOf": [{"maxLength": 200, "type": "string"}, {"type": "null"}], "default": null, "description": "What to say. The robot only has duck sounds; text is mapped to a tone.", "title": "Text"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}, {"name": "gaze", "core": false, "description": "", "params_schema": {"additionalProperties": false, "properties": {"direction": {"default": "center", "description": "Where to point the head.", "enum": ["center", "left", "right", "up", "down"], "title": "Direction", "type": "string"}, "bearing_deg": {"anyOf": [{"maximum": 90, "minimum": -90, "type": "number"}, {"type": "null"}], "default": null, "description": "Optional exact bearing (+ = left). Overrides direction.", "title": "Bearing Deg"}}, "type": "object"}, "safety_class": "safe", "timeout_s": null}], "preconditions": {"move": ["standing"], "kick": ["standing"], "grab": ["standing"], "sit": ["not_fallen"], "stand": ["not_fallen"], "gaze": ["not_fallen"]}, "safety_authority": {"native": "robotd_deadman", "deadman": true, "heartbeat_hz": 2.0}, "frame": {"reference": "body", "note": ""}, "limits": {"max_vx": 0.3, "max_vy": 0.2, "max_wz": 1.5}, "backend": "sim2d", "blurb": "a small biped duck robot (25 cm, 800 g)", "extras": {}}, "dry_run": false, "contract": {"duck": 0, "name": "find-and-kick", "description": "Search the area for a ball, walk to it, kick it.", "author": "rok", "verbs": {"allow": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"], "confirm": []}, "budgets": {"max_steps": 40, "max_minutes": 5.0, "max_llm_calls": 40}, "success": ["Ball displaced more than 0.3 m in sim, or human confirms the kick landed."], "abort_when": ["Battery below 15%", "Same verb fails 3 times in a row"], "persona": "Determined and cheerful. Quack once when you succeed.", "providers": ["fake", "anthropic", "openai", "gemini", "grok"], "learned_verbs": [], "flock": null, "requires": [], "robots": null}, "system_prompt": "You are the brain of a small biped duck robot (25 cm, 800 g). You are a high-level pilot:\nyou choose ONE verb per turn; the robot's own controllers handle balance and gait, and composite\nverbs like `walk_to` close their own loops on the camera. Do not micro-manage.\n\n## Rules (enforced by the executor — not optional)\n- Call exactly one tool per turn. Never zero, never two.\n- Only these verbs are allowed: search_scan, walk_to, kick, quack, get_frame, stop. Anything else is refused.\n- Budgets: 40 steps, 5 minutes, 40 LLM calls. The run stops when any is hit.\n- Verbs marked confirm (none) ask a human before running.\n- When a success criterion is met, call `declare_success`. If the task is impossible, call `declare_failure`.\n\n## Success criteria\n- Ball displaced more than 0.3 m in sim, or human confirms the kick landed.\n\n## Abort conditions you must respect yourself\n- (none beyond the enforced ones)\n\n## Verbs\n- `search_scan`: Rotate in steps, looking for a target. Returns where it was seen.\n- `walk_to`: Walk toward a detected target and stop at a distance. Closes the loop on the camera itself.\n- `kick`: Kick forward with one leg. Only connects if the ball is < 0.3 m away and roughly ahead.\n- `quack`: Make a duck sound, optionally with text (mapped to a tone).\n- `get_frame`: Capture a camera frame and report what is detected in it.\n- `stop`: Stop moving immediately (zero velocity). Always allowed.\n\n## Persona\nDetermined and cheerful. Quack once when you succeed.\n\n## What you remember from earlier runs on this robot\nYour most recent runs on this robot (newest first):\n- [2026-09-03] find-and-kick: success — Ball moved 0.38 m after kick, meeting the success criteria. (4 steps) · search_scan: ball found: ball at bearing 5° left ~1.00 m (after 5 turn steps); walk_to: reached the ball: ~0.24 m away, bearing +0°; kick: kicked with right leg, ball moved 0.38 m; quack: quacked [chirp]\n\nCall `remember` (one short sentence) when you learn something worth keeping for next time:\nwhere an object usually is, which strategy worked, what to avoid. It is free: it moves\nnothing and costs no step. Do not save what is already listed above.\n\nYou are in the built-in 2D simulator: a cartoon top-down world. Distances are metres, the arena is about 2 m across, and the ball is orange.\n\n## Task file: find-and-kick — Search the area for a ball, walk to it, kick it.\n\n# Task\n\nYou are piloting a small biped duck robot. Find the ball and kick it.\n\n## Strategy\n\n1. `search_scan` to locate the ball. It rotates in steps and reports detections.\n2. `walk_to` the ball and stop about 0.25 m away. `walk_to` closes the approach loop\n itself using the camera — you do not need to micro-manage it.\n3. `kick`. The result tells you whether the ball moved and by how much.\n4. Verify with a fresh frame or the kick result. If the ball has not moved at least\n 0.3 m, `walk_to` it again and retry the kick.\n5. When the ball has moved ≥ 0.3 m, `quack` once and declare success.\n\n## Notes\n\n- A kick only connects if the ball is close (< 0.3 m) and roughly in front of you.\n If the kick reports \"missed\", reposition with `walk_to` before kicking again.\n- If `search_scan` finds nothing twice in a row, declare failure rather than looping.\n\n## Memory\n\nBefore you declare success or failure, call `remember` once with one short fact a future\nrun can use: where the ball was when you found it (bearing and rough distance from where\nyou started) or what made the kick connect or miss. Skip it if the prompt already\nremembers the same thing. It costs no step.\n\n## If you cannot call tools natively\nReply with exactly one JSON object and nothing else, in this shape:\n{\"name\": \"\", \"arguments\": {}}\n", "tools": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop", "declare_success", "declare_failure", "remember"], "memory": {"robot": "microduck:sim2d", "path": "../memory-qwen2/microduck-sim2d.jsonl", "notes": 0, "episodes": 1}} +{"t": 0.018, "kind": "frame", "path": "frames/0000.png", "caption": "step 0: start"} +{"t": 0.019, "kind": "observation", "step": 0, "text": "[step 0/40 · step 0/40, llm calls 0/40, 0.0/5 min]\nstate: posture=standing policy=stand battery=100% pose=(0.02, -0.09, -0.82 rad)\ncamera: nothing detected\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 0.0, "x": 0.022898610883165915, "y": -0.09403747811199693, "theta": -0.8226747951461474, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 100.0, "holding": false, "extras": {"sim_time": 0.0, "ball": {"x": -0.188, "y": 0.731, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [], "last_result": null, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 6.924, "kind": "llm", "step": 0, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "708275807", "name": "search_scan", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 2652, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 6.926, "kind": "frame", "path": "frames/0001.png", "caption": "search_scan 0"} +{"t": 6.933, "kind": "frame", "path": "frames/0002.png", "caption": "search_scan 1"} +{"t": 6.939, "kind": "frame", "path": "frames/0003.png", "caption": "search_scan 2"} +{"t": 6.945, "kind": "frame", "path": "frames/0004.png", "caption": "search_scan 3"} +{"t": 6.947, "kind": "verb", "step": 1, "name": "search_scan", "canonical": "search_scan", "params": {"target": "ball"}, "ok": true, "summary": "ball found: ball at bearing 14° left ~0.83 m (after 3 turn steps)", "data": {"detections": [{"label": "ball", "cx": 0.376953125, "cy": 0.5390625, "area": 0.002838134765625, "confidence": 0.9650000000000001, "bearing_deg": 13.8, "est_distance_m": 0.832}], "steps": 3}} +{"t": 6.948, "kind": "frame", "path": "frames/0005.png", "caption": "step 1: search_scan"} +{"t": 6.95, "kind": "observation", "step": 1, "text": "[step 1/40 · step 1/40, llm calls 1/40, 0.0/5 min]\nstate: posture=standing policy=stand battery=100% pose=(0.02, -0.09, 1.58 rad)\ncamera: ball at bearing 14° left ~0.83 m\nlast verb `search_scan`: ok — ball found: ball at bearing 14° left ~0.83 m (after 3 turn steps)\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 2.6999999999999984, "x": 0.022898610883165915, "y": -0.09403747811199693, "theta": 1.5799555297252343, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.946, "holding": false, "extras": {"sim_time": 2.7, "ball": {"x": -0.188, "y": 0.731, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.376953125, "cy": 0.5390625, "area": 0.002838134765625, "confidence": 0.9650000000000001, "bearing_deg": 13.8, "est_distance_m": 0.832}], "last_result": {"verb": "search_scan", "ok": true, "summary": "ball found: ball at bearing 14° left ~0.83 m (after 3 turn steps)", "data": {"detections": [{"label": "ball", "cx": 0.376953125, "cy": 0.5390625, "area": 0.002838134765625, "confidence": 0.9650000000000001, "bearing_deg": 13.8, "est_distance_m": 0.832}], "steps": 3}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 10.167, "kind": "llm", "step": 1, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "421996211", "name": "walk_to", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 2815, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 10.169, "kind": "frame", "path": "frames/0006.png", "caption": "go_to ball"} +{"t": 10.173, "kind": "frame", "path": "frames/0007.png", "caption": "go_to ball"} +{"t": 10.176, "kind": "frame", "path": "frames/0008.png", "caption": "go_to ball"} +{"t": 10.178, "kind": "frame", "path": "frames/0009.png", "caption": "go_to ball"} +{"t": 10.181, "kind": "frame", "path": "frames/0010.png", "caption": "go_to ball"} +{"t": 10.183, "kind": "frame", "path": "frames/0011.png", "caption": "go_to ball"} +{"t": 10.185, "kind": "frame", "path": "frames/0012.png", "caption": "go_to ball"} +{"t": 10.187, "kind": "frame", "path": "frames/0013.png", "caption": "go_to ball"} +{"t": 10.19, "kind": "frame", "path": "frames/0014.png", "caption": "go_to ball"} +{"t": 10.193, "kind": "frame", "path": "frames/0015.png", "caption": "go_to ball"} +{"t": 10.195, "kind": "frame", "path": "frames/0016.png", "caption": "go_to ball"} +{"t": 10.197, "kind": "frame", "path": "frames/0017.png", "caption": "go_to ball"} +{"t": 10.2, "kind": "frame", "path": "frames/0018.png", "caption": "go_to ball"} +{"t": 10.202, "kind": "frame", "path": "frames/0019.png", "caption": "go_to ball"} +{"t": 10.204, "kind": "frame", "path": "frames/0020.png", "caption": "go_to ball"} +{"t": 10.207, "kind": "frame", "path": "frames/0021.png", "caption": "go_to ball"} +{"t": 10.209, "kind": "frame", "path": "frames/0022.png", "caption": "go_to ball"} +{"t": 10.211, "kind": "frame", "path": "frames/0023.png", "caption": "go_to ball"} +{"t": 10.213, "kind": "frame", "path": "frames/0024.png", "caption": "go_to ball"} +{"t": 10.215, "kind": "frame", "path": "frames/0025.png", "caption": "go_to ball"} +{"t": 10.218, "kind": "frame", "path": "frames/0026.png", "caption": "go_to ball"} +{"t": 10.22, "kind": "frame", "path": "frames/0027.png", "caption": "go_to ball"} +{"t": 10.222, "kind": "frame", "path": "frames/0028.png", "caption": "go_to ball"} +{"t": 10.224, "kind": "frame", "path": "frames/0029.png", "caption": "go_to ball"} +{"t": 10.227, "kind": "frame", "path": "frames/0030.png", "caption": "go_to ball"} +{"t": 10.229, "kind": "frame", "path": "frames/0031.png", "caption": "go_to ball"} +{"t": 10.231, "kind": "frame", "path": "frames/0032.png", "caption": "go_to ball"} +{"t": 10.233, "kind": "frame", "path": "frames/0033.png", "caption": "go_to ball"} +{"t": 10.235, "kind": "frame", "path": "frames/0034.png", "caption": "go_to ball"} +{"t": 10.237, "kind": "frame", "path": "frames/0035.png", "caption": "go_to ball"} +{"t": 10.24, "kind": "frame", "path": "frames/0036.png", "caption": "go_to ball"} +{"t": 10.242, "kind": "frame", "path": "frames/0037.png", "caption": "go_to ball"} +{"t": 10.244, "kind": "frame", "path": "frames/0038.png", "caption": "go_to ball"} +{"t": 10.246, "kind": "frame", "path": "frames/0039.png", "caption": "go_to ball"} +{"t": 10.248, "kind": "frame", "path": "frames/0040.png", "caption": "go_to ball"} +{"t": 10.251, "kind": "frame", "path": "frames/0041.png", "caption": "go_to ball"} +{"t": 10.253, "kind": "frame", "path": "frames/0042.png", "caption": "go_to ball"} +{"t": 10.255, "kind": "frame", "path": "frames/0043.png", "caption": "go_to ball"} +{"t": 10.257, "kind": "frame", "path": "frames/0044.png", "caption": "go_to ball"} +{"t": 10.259, "kind": "verb", "step": 2, "name": "walk_to", "canonical": "go_to", "params": {"target": "ball"}, "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.242, "bearing_deg": 0.0, "ticks": 39}} +{"t": 10.26, "kind": "frame", "path": "frames/0045.png", "caption": "step 2: walk_to"} +{"t": 10.261, "kind": "observation", "step": 2, "text": "[step 2/40 · step 2/40, llm calls 2/40, 0.1/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.12, 0.50, 1.84 rad)\ncamera: ball dead ahead ~0.24 m\nlast verb `walk_to`: ok — reached the ball: ~0.24 m away, bearing +0°\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 6.5999999999999845, "x": -0.1242526964459676, "y": 0.49863001147922587, "theta": 1.8404184250668225, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.868, "holding": false, "extras": {"sim_time": 6.6, "ball": {"x": -0.188, "y": 0.731, "present": true}, "ball_displacement_m": 0.0, "last_kick_ball_moved_m": null, "kicks": 0, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.5, "cy": 0.76171875, "area": 0.0335845947265625, "confidence": 1.0, "bearing_deg": -0.0, "est_distance_m": 0.242}], "last_result": {"verb": "walk_to", "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.242, "bearing_deg": 0.0, "ticks": 39}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 13.217, "kind": "llm", "step": 2, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "401399688", "name": "kick", "arguments": {"leg": "right"}}], "usage": {"input_tokens": 2966, "output_tokens": 25}, "stop_reason": "tool_calls"} +{"t": 13.226, "kind": "verb", "step": 3, "name": "kick", "canonical": "kick", "params": {"leg": "right"}, "ok": true, "summary": "kicked with right leg, ball moved 0.15 m", "data": {"ball_moved_m": 0.152}} +{"t": 13.227, "kind": "frame", "path": "frames/0046.png", "caption": "step 3: kick"} +{"t": 13.229, "kind": "observation", "step": 3, "text": "[step 3/40 · step 3/40, llm calls 3/40, 0.1/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.12, 0.50, 1.84 rad)\ncamera: ball at bearing 10° left ~0.39 m\nlast verb `kick`: ok — kicked with right leg, ball moved 0.15 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 8.09999999999998, "x": -0.1242526964459676, "y": 0.49863001147922587, "theta": 1.8404184250668225, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.838, "holding": false, "extras": {"sim_time": 8.1, "ball": {"x": -0.29, "y": 0.844, "present": true}, "ball_displacement_m": 0.152, "last_kick_ball_moved_m": 0.152, "kicks": 1, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.41015625, "cy": 0.64453125, "area": 0.0130767822265625, "confidence": 1.0, "bearing_deg": 10.2, "est_distance_m": 0.387}], "last_result": {"verb": "kick", "ok": true, "summary": "kicked with right leg, ball moved 0.15 m", "data": {"ball_moved_m": 0.152}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 16.244, "kind": "llm", "step": 3, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "283888237", "name": "walk_to", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 3117, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 16.247, "kind": "frame", "path": "frames/0047.png", "caption": "go_to ball"} +{"t": 16.25, "kind": "frame", "path": "frames/0048.png", "caption": "go_to ball"} +{"t": 16.252, "kind": "frame", "path": "frames/0049.png", "caption": "go_to ball"} +{"t": 16.255, "kind": "frame", "path": "frames/0050.png", "caption": "go_to ball"} +{"t": 16.257, "kind": "frame", "path": "frames/0051.png", "caption": "go_to ball"} +{"t": 16.259, "kind": "frame", "path": "frames/0052.png", "caption": "go_to ball"} +{"t": 16.262, "kind": "frame", "path": "frames/0053.png", "caption": "go_to ball"} +{"t": 16.264, "kind": "frame", "path": "frames/0054.png", "caption": "go_to ball"} +{"t": 16.266, "kind": "frame", "path": "frames/0055.png", "caption": "go_to ball"} +{"t": 16.268, "kind": "frame", "path": "frames/0056.png", "caption": "go_to ball"} +{"t": 16.27, "kind": "frame", "path": "frames/0057.png", "caption": "go_to ball"} +{"t": 16.274, "kind": "frame", "path": "frames/0058.png", "caption": "go_to ball"} +{"t": 16.276, "kind": "frame", "path": "frames/0059.png", "caption": "go_to ball"} +{"t": 16.279, "kind": "frame", "path": "frames/0060.png", "caption": "go_to ball"} +{"t": 16.281, "kind": "frame", "path": "frames/0061.png", "caption": "go_to ball"} +{"t": 16.283, "kind": "verb", "step": 4, "name": "walk_to", "canonical": "go_to", "params": {"target": "ball"}, "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.244, "bearing_deg": 0.2, "ticks": 15}} +{"t": 16.284, "kind": "frame", "path": "frames/0062.png", "caption": "step 4: walk_to"} +{"t": 16.285, "kind": "observation", "step": 4, "text": "[step 4/40 · step 4/40, llm calls 4/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.18, 0.63, 2.03 rad)\ncamera: ball dead ahead ~0.24 m\nlast verb `walk_to`: ok — reached the ball: ~0.24 m away, bearing +0°\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 9.600000000000001, "x": -0.18148846702554064, "y": 0.6260237651230226, "theta": 2.0316710037782926, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.808, "holding": false, "extras": {"sim_time": 9.6, "ball": {"x": -0.29, "y": 0.844, "present": true}, "ball_displacement_m": 0.152, "last_kick_ball_moved_m": 0.152, "kicks": 1, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.498046875, "cy": 0.7578125, "area": 0.033111572265625, "confidence": 1.0, "bearing_deg": 0.2, "est_distance_m": 0.244}], "last_result": {"verb": "walk_to", "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +0°", "data": {"distance_m": 0.244, "bearing_deg": 0.2, "ticks": 15}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 19.264, "kind": "llm", "step": 4, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "961913944", "name": "kick", "arguments": {"leg": "right"}}], "usage": {"input_tokens": 3268, "output_tokens": 25}, "stop_reason": "tool_calls"} +{"t": 19.276, "kind": "verb", "step": 5, "name": "kick", "canonical": "kick", "params": {"leg": "right"}, "ok": true, "summary": "kicked with right leg, ball moved 0.18 m", "data": {"ball_moved_m": 0.183}} +{"t": 19.277, "kind": "frame", "path": "frames/0063.png", "caption": "step 5: kick"} +{"t": 19.278, "kind": "observation", "step": 5, "text": "[step 5/40 · step 5/40, llm calls 5/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.18, 0.63, 2.03 rad)\ncamera: ball at bearing 31° left ~0.34 m\nlast verb `kick`: ok — kicked with right leg, ball moved 0.18 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 11.100000000000023, "x": -0.18148846702554064, "y": 0.6260237651230226, "theta": 2.0316710037782926, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.778, "holding": false, "extras": {"sim_time": 11.1, "ball": {"x": -0.47, "y": 0.809, "present": true}, "ball_displacement_m": 0.293, "last_kick_ball_moved_m": 0.183, "kicks": 2, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.197265625, "cy": 0.66796875, "area": 0.01690673828125, "confidence": 1.0, "bearing_deg": 31.2, "est_distance_m": 0.341}], "last_result": {"verb": "kick", "ok": true, "summary": "kicked with right leg, ball moved 0.18 m", "data": {"ball_moved_m": 0.183}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 22.361, "kind": "llm", "step": 5, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "646573368", "name": "walk_to", "arguments": {"target": "ball"}}], "usage": {"input_tokens": 3419, "output_tokens": 26}, "stop_reason": "tool_calls"} +{"t": 22.364, "kind": "frame", "path": "frames/0064.png", "caption": "go_to ball"} +{"t": 22.37, "kind": "frame", "path": "frames/0065.png", "caption": "go_to ball"} +{"t": 22.373, "kind": "frame", "path": "frames/0066.png", "caption": "go_to ball"} +{"t": 22.375, "kind": "frame", "path": "frames/0067.png", "caption": "go_to ball"} +{"t": 22.378, "kind": "frame", "path": "frames/0068.png", "caption": "go_to ball"} +{"t": 22.38, "kind": "frame", "path": "frames/0069.png", "caption": "go_to ball"} +{"t": 22.383, "kind": "frame", "path": "frames/0070.png", "caption": "go_to ball"} +{"t": 22.385, "kind": "frame", "path": "frames/0071.png", "caption": "go_to ball"} +{"t": 22.388, "kind": "frame", "path": "frames/0072.png", "caption": "go_to ball"} +{"t": 22.39, "kind": "frame", "path": "frames/0073.png", "caption": "go_to ball"} +{"t": 22.392, "kind": "frame", "path": "frames/0074.png", "caption": "go_to ball"} +{"t": 22.394, "kind": "frame", "path": "frames/0075.png", "caption": "go_to ball"} +{"t": 22.396, "kind": "verb", "step": 6, "name": "walk_to", "canonical": "go_to", "params": {"target": "ball"}, "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +1°", "data": {"distance_m": 0.244, "bearing_deg": 1.3, "ticks": 12}} +{"t": 22.397, "kind": "frame", "path": "frames/0076.png", "caption": "step 6: walk_to"} +{"t": 22.398, "kind": "observation", "step": 6, "text": "[step 6/40 · step 6/40, llm calls 6/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.26, 0.69, 2.60 rad)\ncamera: ball dead ahead ~0.24 m\nlast verb `walk_to`: ok — reached the ball: ~0.24 m away, bearing +1°\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 12.30000000000004, "x": -0.2585395167483543, "y": 0.688311967025222, "theta": 2.598946723755427, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.754, "holding": false, "extras": {"sim_time": 12.3, "ball": {"x": -0.47, "y": 0.809, "present": true}, "ball_displacement_m": 0.293, "last_kick_ball_moved_m": 0.183, "kicks": 2, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.48828125, "cy": 0.755859375, "area": 0.033111572265625, "confidence": 1.0, "bearing_deg": 1.3, "est_distance_m": 0.244}], "last_result": {"verb": "walk_to", "ok": true, "summary": "reached the ball: ~0.24 m away, bearing +1°", "data": {"distance_m": 0.244, "bearing_deg": 1.3, "ticks": 12}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 25.397, "kind": "llm", "step": 6, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "826271284", "name": "kick", "arguments": {"leg": "right"}}], "usage": {"input_tokens": 3570, "output_tokens": 25}, "stop_reason": "tool_calls"} +{"t": 25.406, "kind": "verb", "step": 7, "name": "kick", "canonical": "kick", "params": {"leg": "right"}, "ok": true, "summary": "kicked with right leg, ball moved 0.44 m", "data": {"ball_moved_m": 0.444}} +{"t": 25.407, "kind": "frame", "path": "frames/0077.png", "caption": "step 7: kick"} +{"t": 25.409, "kind": "observation", "step": 7, "text": "[step 7/40 · step 7/40, llm calls 7/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.26, 0.69, 2.60 rad)\ncamera: ball at bearing 20° left ~0.66 m\nlast verb `kick`: ok — kicked with right leg, ball moved 0.44 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 13.800000000000061, "x": -0.2585395167483543, "y": 0.688311967025222, "theta": 2.598946723755427, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.724, "holding": false, "extras": {"sim_time": 13.8, "ball": {"x": -0.914, "y": 0.822, "present": true}, "ball_displacement_m": 0.732, "last_kick_ball_moved_m": 0.444, "kicks": 3, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.322265625, "cy": 0.5625, "area": 0.00457763671875, "confidence": 1.0, "bearing_deg": 19.6, "est_distance_m": 0.655}], "last_result": {"verb": "kick", "ok": true, "summary": "kicked with right leg, ball moved 0.44 m", "data": {"ball_moved_m": 0.444}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 28.342, "kind": "llm", "step": 7, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "515048734", "name": "get_frame", "arguments": {}}], "usage": {"input_tokens": 3721, "output_tokens": 23}, "stop_reason": "tool_calls"} +{"t": 28.344, "kind": "frame", "path": "frames/0078.png", "caption": "observe"} +{"t": 28.348, "kind": "verb", "step": 8, "name": "get_frame", "canonical": "observe", "params": {}, "ok": true, "summary": "frame captured; ball at bearing 20° left ~0.66 m", "data": {"detections": [{"label": "ball", "cx": 0.322265625, "cy": 0.5625, "area": 0.00457763671875, "confidence": 1.0, "bearing_deg": 19.6, "est_distance_m": 0.655}]}} +{"t": 28.35, "kind": "frame", "path": "frames/0079.png", "caption": "step 8: get_frame"} +{"t": 28.351, "kind": "observation", "step": 8, "text": "[step 8/40 · step 8/40, llm calls 8/40, 0.2/5 min]\nstate: posture=standing policy=stand battery=100% pose=(-0.26, 0.69, 2.60 rad)\ncamera: ball at bearing 20° left ~0.66 m\nlast verb `get_frame`: ok — frame captured; ball at bearing 20° left ~0.66 m\nChoose exactly one tool.", "has_image": false, "features": {"state": {"t": 13.800000000000061, "x": -0.2585395167483543, "y": 0.688311967025222, "theta": 2.598946723755427, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.724, "holding": false, "extras": {"sim_time": 13.8, "ball": {"x": -0.914, "y": 0.822, "present": true}, "ball_displacement_m": 0.732, "last_kick_ball_moved_m": 0.444, "kicks": 3, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}, "detections": [{"label": "ball", "cx": 0.322265625, "cy": 0.5625, "area": 0.00457763671875, "confidence": 1.0, "bearing_deg": 19.6, "est_distance_m": 0.655}], "last_result": {"verb": "get_frame", "ok": true, "summary": "frame captured; ball at bearing 20° left ~0.66 m", "data": {"detections": [{"label": "ball", "cx": 0.322265625, "cy": 0.5625, "area": 0.00457763671875, "confidence": 1.0, "bearing_deg": 19.6, "est_distance_m": 0.655}]}}, "allowed": ["search_scan", "walk_to", "kick", "quack", "get_frame", "stop"]}} +{"t": 32.762, "kind": "llm", "step": 8, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "text": null, "tool_calls": [{"id": "573502212", "name": "declare_success", "arguments": {"reason": "Ball moved 0.44 m after two kicks, meeting the success criteria."}}], "usage": {"input_tokens": 3875, "output_tokens": 42}, "stop_reason": "tool_calls"} +{"t": 32.762, "kind": "declare", "step": 8, "outcome": "success", "reason": "Ball moved 0.44 m after two kicks, meeting the success criteria."} +{"t": 32.763, "kind": "run_end", "duck": "find-and-kick", "outcome": "success", "reason": "Ball moved 0.44 m after two kicks, meeting the success criteria.", "steps": 8, "llm_calls": 9, "elapsed_s": 13.8, "usage": {"input_tokens": 29403, "output_tokens": 244}, "provider": "lmstudio", "model": "qwen/qwen2.5-coder-14b", "transport": "sim2d", "robot": "microduck", "dry_run": false, "final_state": {"t": 13.800000000000061, "x": -0.2585395167483543, "y": 0.688311967025222, "theta": 2.598946723755427, "policy": "stand", "posture": "standing", "fallen": false, "battery_percent": 99.724, "holding": false, "extras": {"sim_time": 13.8, "ball": {"x": -0.914, "y": 0.822, "present": true}, "ball_displacement_m": 0.732, "last_kick_ball_moved_m": 0.444, "kicks": 3, "quacks": 0, "head_yaw_deg": 0.0, "people": [{"x": -0.603, "y": -0.717}]}}} diff --git a/docs/local-llms.md b/docs/local-llms.md index e2025ff..1d86a26 100644 --- a/docs/local-llms.md +++ b/docs/local-llms.md @@ -138,15 +138,35 @@ What the page has and has not been run against is in [web/README.md](../web/READ ## Honest notes -- Which local model pilots the duck well is an open question we have not measured. The +- Which local model pilots the duck well is an open question we have barely measured. The loop was designed so that a weak planner degrades the task, never the robot's balance. - There is exactly one data point, and it is not ours: the contributor who built memory - between runs ran `find-and-kick` against **Qwen 2.5 Coder 14B on LM Studio**, seeds 5 and - 6, both successes with memory read and written, and reported that the model ignored a - memory hint sitting in the system prompt but followed the same instruction once it was a - numbered step in the `.duck` body. That is one model, one machine, two seeds, and no - transcript in this repository. If you run one, please share the transcript in a - Discussion: it is the cheapest way to make this section shorter. + Two transcripts are published here and they are not ours: the contributor who built + memory between runs ran `find-and-kick` against **Qwen 2.5 Coder 14B on LM Studio** + (Apple M2 Pro, 2026-09-03) and they are in [`assets/transcripts/`](assets/transcripts/). + They are two runs out of more than two: [docs/design/memory.md](design/memory.md) + records the same model reading the memory block and never writing to it *across four + runs* before `remember` was moved into the numbered strategy. These two were kept, so + read them as a selection rather than as the sample: + + | transcript | seed | outcome | steps | LLM calls | tokens in + out | text fallbacks | what it shows | + |---|---|---|---|---|---|---|---| + | [`…seed6-memory-read.jsonl`](assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed6-memory-read.jsonl) | 6 | success | 8 | 9 | 29,403 + 244 | 0 | the system prompt carries an earlier run's episode under *What you remember*; the model never calls `remember`; two kicks fall short before the third connects | + | [`…seed5-remember.jsonl`](assets/transcripts/qwen2.5-coder-14b-lmstudio-find-and-kick-seed5-remember.jsonl) | 5 | success | 4 | 6 | 17,939 + 265 | 0 | the `.duck` body now says `remember` in strategy step 5; after the kick the model returns `remember`, `quack` and `declare_success` in one response, the loop keeps the first (a fact from the verb results) and marks `multiple_tool_calls`, and the other two arrive one per turn after | + + These two are not a chain, and nothing here should be read as one: they ran against + different memory directories (`memory-qwen3` and `memory-qwen2`), seed 5 started from an + empty memory block, and the episode seed 6 remembers was written by a run that is not in + this repository. Seed 5 shows the write and seed 6 shows the read. Neither shows the + other's half. + + Every turn was a native tool call, none needed the JSON text fallback. The simulator + clock (`elapsed_s` in `run_end`, which is what the budget counts on `sim2d`) says 14 s and + 11 s; the transcript timestamps say 33 s and 29 s of wall clock, three to nine seconds per + LLM call. What they cannot show: anything about another model, another machine, or a + harder task than the starter duck, and neither one shows a note surviving from the run + that wrote it into the run that reads it. If you run one, please + share the transcript in a Discussion or a PR into that folder: it is the cheapest way + to make this section shorter. - The cloud providers keep their stricter settings (`tool_choice="required"`, `parallel_tool_calls=False`). Only the local presets use the relaxed ones. - Ollama, vLLM, llama.cpp and LM Studio evolve quickly. If a flag above is stale, open an diff --git a/docs/memory.md b/docs/memory.md index c3a7416..95408ca 100644 --- a/docs/memory.md +++ b/docs/memory.md @@ -84,7 +84,7 @@ shows every fact it saved. Telling the model about `remember` in the prompt is not enough for a small local model: in the runs this feature's contributor did, Qwen 2.5 Coder 14B read the memory block and never -wrote to it. What works is +wrote to it (both cases are in [`assets/transcripts/`](assets/transcripts/)). What works is putting the call **inside the numbered strategy** of the `.duck` body, right before the declaration (`5. When the ball has moved ≥ 0.3 m, \`remember\` where you found the ball, \`quack\` once and declare success.`), plus a short *Memory* section saying what is worth