From e82000bc1bdc0d9f7d626744a621b8f94c45fe0d Mon Sep 17 00:00:00 2001 From: Ori Nachum Date: Mon, 10 Aug 2026 00:14:41 +0300 Subject: [PATCH] fix: three surfaces still named the checkpoint 0.54.9 demoted (0.55.1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The multimodal `cortex` promotion swapped the served id to `unsloth/Qwen3.6-27B-NVFP4` and left `sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP` behind as a text-only candidate. Three surfaces kept describing the demoted one as what this box serves: - `lobes whoami`'s `_DEFAULT_MODEL` fallback — the id it prints when there is no scaffold or `VLLM_SERVED_NAME` is unset — disagreed with the gateway's own `_DEFAULT_PRIMARY`, and `whoami`'s answer was the wrong one: a caller pinning what it printed gets `model_not_found`. - `AGENTS.md`, the deployed agent's actual `acp` system prompt, described a text-only runtime with a grafted MTP head. An agent that does not know it can see keeps referring vision away, so the paragraph now states the image/video intake through the checkpoint's own ViT. - `README.md` taught the addressing pattern that just broke — an example pinning the retired raw id. It now addresses the stable `cortex` role name, and the neighbouring comment no longer claims an unknown model falls back to the primary (only a *missing* `model` field does; an unknown id 404s). Also lands eidetic recall bookkeeping and one new public record, `spark-proxy-advert-refresh-20260805`. Docs and fallback constants only — no runtime routing changed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Eq2ZEyb6PV9ntkSzfsuRsm --- .eidetic/memory/lobes__public.jsonl | 27 ++++++++++++++------------- AGENTS.md | 13 ++++++++----- CHANGELOG.md | 15 +++++++++++++++ README.md | 8 +++++--- lobes/cli/_commands/whoami.py | 6 +++++- pyproject.toml | 2 +- uv.lock | 2 +- 7 files changed, 49 insertions(+), 24 deletions(-) diff --git a/.eidetic/memory/lobes__public.jsonl b/.eidetic/memory/lobes__public.jsonl index 7154a6e3..069fa757 100644 --- a/.eidetic/memory/lobes__public.jsonl +++ b/.eidetic/memory/lobes__public.jsonl @@ -32,13 +32,13 @@ {"id": "senses-gemma-cannot-hear-2026-07-09", "hash": "93dee119b623b8d35a77422ef56604c09119e3fb72196f1433f91c04a8514dbf", "content": "lobes `senses` role (coolthor/gemma-4-12B-it-NVFP4A16 on vLLM 0.23.1rc1.dev672+g93d8f834d) CANNOT HEAR \u2014 vLLM silently DROPS `input_audio` OpenAI content parts (filed as issue #101). Proof by prompt-token accounting via model=multimodal: text-only = 15 prompt_tokens; text+image (96x96 solid PNG) = 273 (+258, and the model correctly answers \"Red\"/\"Blue\"); text+audio (0.68s WAV @24kHz from the rig's own Chatterbox) = 34 (+19 placeholder tokens) with empty content; same clip resampled to 16kHz = 48 tokens and the model replies \"I cannot hear any audio because you haven't provided a file or a link\". Sample rate is NOT the cause. The checkpoint is fine: AutoConfig shows audio_config + audio_token_id + vision_config. So vLLM's gemma4_unified path wires the vision encoder and not the audio encoder, and DROPS rather than REJECTS the audio part \u2014 caller gets HTTP 200 and a fluent answer that ignored the audio. HOW IT WENT UNNOTICED FOR SO LONG: tests/test_smoke_duo.py's live layer asserted only `assert content.strip()` (HTTP 200 + non-empty) against a 1x1 placeholder PNG and a tiny WAV. That proves the WIRE, not the PERCEPTION. The docs' \"image+text \u2713 / audio+text \u2713\" for the sakamakismile coder checkpoint rest on the same weak check. METHOD THAT WORKS: generate a solid-colour PNG with stdlib zlib+struct, ask \"What single colour fills this image? Answer with one word.\", assert the colour name appears; ALWAYS run a negative control (feed blue, assert \"red\" fails) or the test is vacuous. For audio, synthesize a known word with the rig's own Chatterbox TTS (POST /v1/audio/speech) and assert the transcription contains it. The audio probe is now @pytest.mark.xfail(strict=True) citing #101, so it XPASSes and fails the suite the day audio starts working. NOTE the `stt` role (Parakeet, POST /v1/audio/transcriptions) is a separate purpose-built path and WORKS: tts(\"banana\") -> stt -> \"Banana.\"; tts(\"the quick brown fox\") -> \"The Quick Brown Fox\".", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "investigation", "record_metadata": {"source": "claude-code-session", "topic": "senses-audio", "relates_to": "issues-101-74", "date": "2026-07-09"}, "created": "2026-07-09T09:38:15.311269+00:00", "last_recall": "2026-07-24T18:41:34.170386+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "lobes-pressure-swap-occupancy-sticky-2026-07-09", "hash": "7f666a9ea25527288cadc52f8b87e5499164ea353ffed40d2e3eda9aa7868e6b", "content": "lobes pressure policy sheds 100% of generate traffic FOREVER on a box whose swap has ever filled (filed as issue #100). lobes/runtime/_pressure.py computes swap_used_percent = (SwapTotal - SwapFree)/SwapTotal from /proc/meminfo \u2014 an OCCUPANCY metric. Swap occupancy is STICKY: the kernel does not reclaim swapped pages just because RAM freed, so once it crosses the 75% default threshold it stays there until swapoff/swapon. On the DGX Spark GB10 (2026-07-09): swap 16322/16383 MB = 99.6% occupancy, BUT `vmstat` si=0 so=0 (nothing being swapped) and /proc/pressure/memory shows `full avg10=0.00 avg60=0.00` (ZERO memory stall), with 26 GB RAM available. Since #88 (0.37.0) the policy SHEDS with 429 server_busy instead of degrading, so every cortex/senses request 429s while `X-Lobes-Override: 1` still returns 200. The rig had been fine on gateway 0.36.0 (pre-shed) and broke the instant it was upgraded to 0.39.0 \u2014 the wire had been armed for weeks. WORKAROUND (documented, but it DISABLES the policy rather than fixing it): set LOBES_SWAP_DEGRADED_THRESHOLD=100 and LOBES_IOWAIT_DEGRADED_THRESHOLD=100 in the deployment .env and recreate the gateway. Note LOBES_IOWAIT_DEGRADED_THRESHOLD=100 was ALREADY set by a previous operator for the same class of reason (0.36.2 CHANGELOG documents \"phantom high iowait on an idle disk, e.g. the DGX Spark GB10\") \u2014 half the policy had already been switched off, which is the tell that the METRIC is wrong, not the threshold. RIGHT FIX: read PSI (/proc/pressure/memory, /proc/pressure/io avg10) or swap RATE (pswpin/pswpout deltas from /proc/vmstat), not occupancy; keep occupancy at most as a secondary AND-condition. Also: a gateway that has shed 100% of requests for an hour is misconfigured, not busy \u2014 nothing currently warns.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "pressure-policy", "relates_to": "issues-100-85-86-88", "date": "2026-07-09"}, "created": "2026-07-09T09:38:15.369909+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "lobes-fleet-frozen-image-pin-2026-07-09", "hash": "08f35f73d1ad1f81fc3dd0a6d348a2d8f7f6c2ef4581caa92dfba6c7a56b14c5", "content": "lobes-cli fleet deployments FREEZE their lobes version and nothing bumps it (issue #99). Dockerfile.gateway/realtime/stt/chatterbox all run: pip install --no-cache-dir lobes-cli==${MODEL_GEAR_VERSION}. `lobes init` writes MODEL_GEAR_VERSION into .env ONCE at scaffold time and no verb ever rewrites it, so `docker compose up -d --build gateway` faithfully reinstalls the OLD pinned version. On 2026-07-09 the reference rig ran THREE lobes versions at once: gateway 0.36.0 (image built 07-03), realtime 0.34.1 (07-02), stt/chatterbox 0.34.1-era (07-02) \u2014 against a CLI at 0.39.0, with 0.36.1/0.36.2/0.37.0/0.38.0/0.39.0 ALL published on PyPI. CONSEQUENCE: three bugs were reported against lobes CODE that were merely undeployed fixes. (a) #92 'advertised endpoint :8000 404s' \u2014 reachable_origin shipped in 0.38.0/PR#90 on 07-04; the gateway image predated it by one day, so it fell back to _gateway_base_url() = its INTERNAL container port 8000 while the published host port is 8001. Worse, host :8000 was an unrelated uvicorn service (reachy-mini-dae), so the advertised endpoint pointed at a foreign daemon. (b) the stt sidecar answered /v1/health/ready with 200 status=ready while its CUDA context was poisoned and transcription 500d \u2014 #89 (07-04) had replaced the old liveness-only handler with a model_loaded+cuda_ok probe, but the stt image was built 07-02. (c) chatterbox likewise. CHEAP FIX (no model reload): edit ~/.lobes/.env MODEL_GEAR_VERSION=, then from ~/.lobes run: docker compose -f docker-compose.yml -f docker-compose.audio.yml up -d --build --no-deps gateway realtime stt chatterbox. GOTCHA 1: the compose PROJECT is named lobes (from the dir basename) while containers are named model-gear-* via container_name \u2014 check the com.docker.compose.project label before invoking compose or you create a duplicate project. GOTCHA 2: AUDIO_URL reaches the gateway ONLY via docker-compose.audio.yml, so a gateway recreated without the -f audio overlay silently loses the audio lane (issue #96). GOTCHA 3: a from-source checkout can be AHEAD of PyPI, in which case the gateway image cannot be built from the pin at all \u2014 the dev path needs a locally built wheel COPYd in.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "fleet-version-pin", "relates_to": "issues-99-92-89-96", "date": "2026-07-09"}, "created": "2026-07-09T09:38:41.844728+00:00", "last_recall": "2026-07-24T18:41:34.170386+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "unified-memory-budget-is-measured-truth-2026-07-14", "hash": "b6258062566bba19a3de9fad95bb4eba773be9b6b233ae69267473f566b2cf6f", "content": "Brain-shapes live validation (#113, 2026-07-14) PROVED shape budgets on unified-memory boxes are MEASURED truths, not arithmetic. vLLM checks free-at-boot >= util*total: GB10 spark-lobe cortex REFUSED util 0.60 (needs 73.01 GiB; only 59.35/121.69 GiB free \u2014 host OS, eidetic/nova stacks, page cache share the pool) -> shipped 0.44 (reclaim-sum 0.30+0.14) which boots healthy at FULL native 262144 context: KV pool 888,946 tokens, 3.39x concurrency (hybrid Mamba Qwen3.6 = cheap long-context KV). Thor thor-lobe senses REFUSED even the 0.44 reclaim-sum (needs 54.04; only 38.44/122.82 free \u2014 heavier host workloads) -> shipped 0.30 (dropped cortex share only) at full 131072: KV pool 1,418,554 tokens, 10.82x concurrency (Gemma sliding-window layers cap long-context KV). Both PASS transcripts in docs/evidence/. LESSON: the acceptance run (scripts/accept-shape.sh) is what validates a shape budget on a box; declared reclaim values are hypotheses until a physical boot confirms.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "shape-budgets-measured", "relates_to": "issue-113-112", "date": "2026-07-14"}, "created": "2026-07-14T09:47:24.654680+00:00", "last_recall": "2026-07-31T13:22:52.104978+00:00", "recall_count": 5.0, "links": ["issue-112-one-lobe-per-box-specced"], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "unified-memory-budget-is-measured-truth-2026-07-14", "hash": "b6258062566bba19a3de9fad95bb4eba773be9b6b233ae69267473f566b2cf6f", "content": "Brain-shapes live validation (#113, 2026-07-14) PROVED shape budgets on unified-memory boxes are MEASURED truths, not arithmetic. vLLM checks free-at-boot >= util*total: GB10 spark-lobe cortex REFUSED util 0.60 (needs 73.01 GiB; only 59.35/121.69 GiB free \u2014 host OS, eidetic/nova stacks, page cache share the pool) -> shipped 0.44 (reclaim-sum 0.30+0.14) which boots healthy at FULL native 262144 context: KV pool 888,946 tokens, 3.39x concurrency (hybrid Mamba Qwen3.6 = cheap long-context KV). Thor thor-lobe senses REFUSED even the 0.44 reclaim-sum (needs 54.04; only 38.44/122.82 free \u2014 heavier host workloads) -> shipped 0.30 (dropped cortex share only) at full 131072: KV pool 1,418,554 tokens, 10.82x concurrency (Gemma sliding-window layers cap long-context KV). Both PASS transcripts in docs/evidence/. LESSON: the acceptance run (scripts/accept-shape.sh) is what validates a shape budget on a box; declared reclaim values are hypotheses until a physical boot confirms.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "shape-budgets-measured", "relates_to": "issue-113-112", "date": "2026-07-14"}, "created": "2026-07-14T09:47:24.654680+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 5.5, "links": ["issue-112-one-lobe-per-box-specced"], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "testpypi-dev-lane-gotchas-2026-07-14", "hash": "1e633b09f4d35c71c04405fa29e60426dc7193baffb9ac12e2a6fc81890b8357", "content": "lobes-cli dev lane (deploy an unreleased branch gateway via TestPyPI .devN, built 2026-07-14 for #113 wave-4): THREE gotchas cost a cycle each. (1) Docker exports build ARGs as env vars during RUN, and pip natively reads PIP_EXTRA_INDEX_URL from env \u2014 an ARG by that name silently poisons EVERY pip call in the layer, not just the intended one; the ARG must be named something pip ignores (LOBES_DEV_INDEX_URL). (2) A plain extra-index against TestPyPI lets name-squats outrank real packages by version (TestPyPI 'FASTAPI-1.0' beat real fastapi 0.115.x and its broken sdist killed the build); the robust pattern is TWO-STEP: pip install --no-deps --index-url lobes-cli==X.devN, THEN a normal PyPI install of the full requirement \u2014 pip sees the exact pin satisfied and resolves only deps from real PyPI. (3) THREE images pin lobes-cli (gateway, realtime, chatterbox) \u2014 dev-lane parity must cover all of them or fleet up dies on the Nth build. Also: ${VAR:+VAR=val} cmd is NOT an env prefix after expansion (words from expansion are never parsed as assignments \u2014 exit 127); use env(1). PR CI publishes 0.X.Y.dev to TestPyPI automatically; the #99 skew check accepts a .devN whose public base equals the CLI version.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "dev-lane-testpypi", "relates_to": "issue-113", "date": "2026-07-14"}, "created": "2026-07-14T09:47:24.657978+00:00", "last_recall": "2026-07-20T16:59:14.575537+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "issue-81-cortex-senses-roles-specced", "hash": "37fc19bb17bdcac85e4d013d2815fafeb6d8a5bbe624f01c592feb3aedaed6de", "content": "Issue #81 (expose full Colleague stack as cortex/senses role-based lobes) specced via /think 2026-07-03 -> docs/specs/2026-07-03-lobes-exposes-the-full-colleague-runtime-stack-as.md. USER-confirmed FULL-#81 scope: six first-class roles cortex/senses/stt/tts/embedder/reranker; REBALANCE cortex(Qwen3.6-27B-MTP)=128K (up from fleet's 64K), senses(Gemma4-12B)=32K (down from 128K), util retuned, pressure degrades cortex->minor (senses is a distinct capability not a rung); 'lobes capabilities'/'lobes endpoint ' + gateway GET /capabilities JSON contract for Colleague; 'lobes up ' + 'colleague-stack' profile; per-role RUNTIME metrics only (Colleague owns task-quality); cortex-only-vs-cortex+senses bench profiles. KEY DECISIONS: (c24) ROLE LAYER not rename -- cortex->primary, senses->multimodal via catalog.TIER_ROLE + mirrors (_pressure_policy._TIER_ROLE, _tier_request); internal service/env/container names (vllm-primary/vllm-multimodal, PRIMARY_*/MULTIMODAL_*) NOT renamed; main|multimodal|hard|normal kept as back-compat aliases. (c25) contract transport = CLI 'lobes capabilities --json' AND gateway GET /capabilities. (c26) fix FLEET_CONTAINERS gap. 'brain' FORBIDDEN as role name/alias. Parked (nonblocking): exact utils, canonical responsibilities word-lists, 'lobes up' verb shape, whether colleague-stack bundles audio overlay. Next leg: /spec-to-plan.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "cortex-senses-roles", "relates_to": "issue-81", "date": "2026-07-03"}, "created": "2026-07-03T08:22:08.845819+00:00", "last_recall": "2026-07-31T13:22:44.488792+00:00", "recall_count": 5.0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "issue-112-one-lobe-per-box-specced", "hash": "e35e355dbbaf1c8541373e807e911db0508ef5b92ed565db1a76fe83c42225e9", "content": "Issue #112 (mesh-brain end-state: one lobe per box) specced via /think 2026-07-14 -> docs/specs/2026-07-14-lobes-serves-the-mesh-brain-end-state-one-lobe-per.md (branch spec/112-one-lobe-per-box). USER DECISIONS: (1) the shape axis is BACKWARD COMPATIBLE and MIXABLE \u2014 many specialized machines (some cloud if you want), some multi-role machines, or any mix; machine-as-brain stays default; one-lobe shapes are the far end of the axis, not a mandate. (2) Cross-box reachability = DIRECT + HONEST REFERRAL: no data-plane proxying; consumers dial each box directly (as the Culture mesh does today); with opt-in peer config, a box's capabilities and role_infeasible 404s NAME the peer hosting an absent role; #92 invariant holds. (3) CHEAP GEARS CO-RESIDE EVERYWHERE: 'one lobe per box' specializes the HEAVY generate lobes only; embedder/reranker/stt/tts stay on every box that wants them (~0.06 util each); consumers keep localhost embed/rerank (eidetic unchanged); Orin takes minor + optionally pooling when validated. (4) Qwen-only Spark stays a FLEET shape (gateway + one heavy vLLM, capabilities surface intact); 'return to 256K-native solo' = full GPU budget + 262144 context as shape overrides, NOT the legacy single-model scaffold. Reference shapes: qwen-only spark @262144 (2.0x co-resident 131072), gemma-only thor senses @131072 (4.0x the 32768 trim), small-model orin DECLARED-BUT-UNVALIDATED until physically booted (#108 rule). Composes on the near-term brain-shapes plan (t1-t8, #113, spec+plan merged, IMPLEMENTATION NOT YET LANDED as of 2026-07-14) \u2014 that plan must land first (stated assumption). Next leg: /spec-to-plan.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "one-lobe-per-box", "relates_to": "issue-112", "date": "2026-07-14"}, "created": "2026-07-14T05:59:15.053474+00:00", "last_recall": "2026-07-31T13:22:52.104978+00:00", "recall_count": 7.5, "links": ["issue-81-cortex-senses-roles-specced"], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "issue-81-cortex-senses-roles-specced", "hash": "37fc19bb17bdcac85e4d013d2815fafeb6d8a5bbe624f01c592feb3aedaed6de", "content": "Issue #81 (expose full Colleague stack as cortex/senses role-based lobes) specced via /think 2026-07-03 -> docs/specs/2026-07-03-lobes-exposes-the-full-colleague-runtime-stack-as.md. USER-confirmed FULL-#81 scope: six first-class roles cortex/senses/stt/tts/embedder/reranker; REBALANCE cortex(Qwen3.6-27B-MTP)=128K (up from fleet's 64K), senses(Gemma4-12B)=32K (down from 128K), util retuned, pressure degrades cortex->minor (senses is a distinct capability not a rung); 'lobes capabilities'/'lobes endpoint ' + gateway GET /capabilities JSON contract for Colleague; 'lobes up ' + 'colleague-stack' profile; per-role RUNTIME metrics only (Colleague owns task-quality); cortex-only-vs-cortex+senses bench profiles. KEY DECISIONS: (c24) ROLE LAYER not rename -- cortex->primary, senses->multimodal via catalog.TIER_ROLE + mirrors (_pressure_policy._TIER_ROLE, _tier_request); internal service/env/container names (vllm-primary/vllm-multimodal, PRIMARY_*/MULTIMODAL_*) NOT renamed; main|multimodal|hard|normal kept as back-compat aliases. (c25) contract transport = CLI 'lobes capabilities --json' AND gateway GET /capabilities. (c26) fix FLEET_CONTAINERS gap. 'brain' FORBIDDEN as role name/alias. Parked (nonblocking): exact utils, canonical responsibilities word-lists, 'lobes up' verb shape, whether colleague-stack bundles audio overlay. Next leg: /spec-to-plan.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "cortex-senses-roles", "relates_to": "issue-81", "date": "2026-07-03"}, "created": "2026-07-03T08:22:08.845819+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 5.5, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "issue-112-one-lobe-per-box-specced", "hash": "e35e355dbbaf1c8541373e807e911db0508ef5b92ed565db1a76fe83c42225e9", "content": "Issue #112 (mesh-brain end-state: one lobe per box) specced via /think 2026-07-14 -> docs/specs/2026-07-14-lobes-serves-the-mesh-brain-end-state-one-lobe-per.md (branch spec/112-one-lobe-per-box). USER DECISIONS: (1) the shape axis is BACKWARD COMPATIBLE and MIXABLE \u2014 many specialized machines (some cloud if you want), some multi-role machines, or any mix; machine-as-brain stays default; one-lobe shapes are the far end of the axis, not a mandate. (2) Cross-box reachability = DIRECT + HONEST REFERRAL: no data-plane proxying; consumers dial each box directly (as the Culture mesh does today); with opt-in peer config, a box's capabilities and role_infeasible 404s NAME the peer hosting an absent role; #92 invariant holds. (3) CHEAP GEARS CO-RESIDE EVERYWHERE: 'one lobe per box' specializes the HEAVY generate lobes only; embedder/reranker/stt/tts stay on every box that wants them (~0.06 util each); consumers keep localhost embed/rerank (eidetic unchanged); Orin takes minor + optionally pooling when validated. (4) Qwen-only Spark stays a FLEET shape (gateway + one heavy vLLM, capabilities surface intact); 'return to 256K-native solo' = full GPU budget + 262144 context as shape overrides, NOT the legacy single-model scaffold. Reference shapes: qwen-only spark @262144 (2.0x co-resident 131072), gemma-only thor senses @131072 (4.0x the 32768 trim), small-model orin DECLARED-BUT-UNVALIDATED until physically booted (#108 rule). Composes on the near-term brain-shapes plan (t1-t8, #113, spec+plan merged, IMPLEMENTATION NOT YET LANDED as of 2026-07-14) \u2014 that plan must land first (stated assumption). Next leg: /spec-to-plan.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "one-lobe-per-box", "relates_to": "issue-112", "date": "2026-07-14"}, "created": "2026-07-14T05:59:15.053474+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 8.5, "links": ["issue-81-cortex-senses-roles-specced"], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "devague-no-move-to-resolve-hard-questions", "hash": "49cf9acb89fd001b4a6c3fd58971c806b7fce31db51869ec9782b83cafcd96b5", "content": "devague gotcha (2026-07-14, adjacent to the known no-unpark gap): there is NO CLI move to resolve a hard question created by 'interrogate --hard-question'. 'devague question --resolve QID' manages a DIFFERENT id-space (.devague/questions/ standalone decisions) and errors 'no such question' for interrogate-created q-ids. A BLOCKING hard question therefore holds converge forever unless (a) its parent claim is rejected (the question dies with the claim) or (b) you hand-edit .devague/frames/.json setting resolved:true on the question object. Correct workflow: record the user's answer as a user-origin decision claim (auto-confirms), then hand-edit resolved:true, and after export annotate the doc's Hard-questions section with 'RESOLVED (user decision, see Decisions): ... -> answer' (export renders questions WITHOUT resolutions, and also dumps rejected claims' hard questions \u2014 trim both). Precedent: the #113 brain-shapes frame did exactly this. Worth an upstream issue on agentculture/devague (resolve-hard-question move).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "devague-hard-questions", "relates_to": "devague", "date": "2026-07-14"}, "created": "2026-07-14T05:59:15.055842+00:00", "last_recall": null, "recall_count": 0, "links": ["devague-export-dumps-rejected-hard-questions", "devague-instruct-flips-confirmed-state"], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "thor-serves-thor-lobe-2026-07-14", "hash": "69e7a5f411a88226fa2172e160bd60b96b470c5cd367d3e77977ee682195d73f", "content": "Deployment change (2026-07-14, PR #121 live test, operator-directed, permanent): the Jetson AGX Thor box now serves the thor-lobe shape \u2014 Gemma senses at max_model_len=131072 / gpu_mem_util=0.30 plus embedder+reranker; the Qwen cortex is DROPPED here (PRIMARY_FEASIBLE=false) with the honest referral PRIMARY_PEER_ORIGIN=http://spark.tail0be7e0.ts.net:8001 (the Spark hosts cortex; its lobes gateway is on port 8001). Callers on the Thor asking for model=main/hard/cortex get 404 role_infeasible with hosted_by \u2014 dial the Spark directly, nothing proxies. Gateway pinned to TestPyPI 0.43.0.dev239 until PR #121 releases 0.43.0 \u2014 re-pin after merge. Previous machine-as-brain deployment preserved at ~/.lobes.pre-accept-20260714T151429Z. The re-scaffold fixed the stale rerank-lane hang (#105/#119): rerank ordering passes in ~1.4s. Full PASS transcript: docs/evidence/2026-07-14-accept-thor-lobe-live-referral-thor.txt. Boot caveat stands: after teardown run sync + drop_caches before fleet up (unified-memory race; 16 restarts without it).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "decision", "record_metadata": {"issue": "#112", "pr": "#121", "box": "thor"}, "created": "2026-07-14T15:52:50.642616+00:00", "last_recall": "2026-07-31T13:22:52.104978+00:00", "recall_count": 5.0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "colleague-320-tool-call-mangle-root-cause", "hash": "1592fa70f1407d998bd87856f4d54472f7779065b4b5a8e75fc44b8d187de8d1", "content": "Colleague #320 (cortex tool calls arrive mangled: name read_file\" + empty arguments, every work item delivers nothing) ROOT-CAUSED 2026-07-14 on the Spark GB10, hypothesis (a) CONFIRMED server-side: a two-way logging proxy captured the RAW gateway/vLLM response during a real colleague work run and the server itself already returns tool_call name=read_file\\\" args={} \u2014 colleague parsing is exonerated. Deterministic minimal repro (temp 0.0): colleague 4K-char system prompt + 14 tools + tool_choice=auto \u2192 the Qwen3.6-27B-MTP cortex drifts off its own chat template and emits a hallucinated \\n\\n[tool_id]{json} format (NOT the qwen3_coder XML its template prescribes); vLLM qwen3 parser salvage mangles it. The long-prompt theory was WRONG \u2014 4K chars suffices; nothing in colleague prompt teaches JSON tool format. FIX PATH: (1) strict:true on function tools arms xgrammar structural-tag constrained decoding (VLLM_ENFORCE_STRICT_TOOL_CALLING defaults True, but with tool_choice=auto it only arms if some tool declares strict:true \u2014 structural_tag_registry.get_model_structural_tag gate); (2) BUT the served vLLM build (0.23.1rc1.dev672+g93d8f834d) hardcodes reasoning=False at the call site (parser/abstract_parser.py _apply_structural_tag \u2192 get_structural_tag(request, reasoning=False)), so with thinking on, the grammar cannot accept the special token (id 248069) \u2192 FSM failure \u2192 HTTP 500 (grammar rejected tokens [,\\n\\n,,\\n]); (3) strict + chat_template_kwargs enable_thinking:false returns a CLEAN structured call (read_file {path: calc.py}) \u2014 proven live, MTP spec-decode active (77.8% draft acceptance) so constrained decoding + MTP coexist; (4) reasoning=True IS supported by this build xgrammar builtin (sequence: any_text \u2192 \u2192 \\n\\n \u2192 triggered tags) \u2014 only the call site never passes it. Candidate lobes fix: tiny --tool-parser-plugin subclassing Qwen3EngineToolParser overriding get_structural_tag to pass reasoning=True when the request has thinking enabled (ToolParserManager.import_tool_parser exists in the image); then colleague sends strict:true (or lobes gateway injects it, config-gated). All 14 colleague tool schemas compiled fine under xgrammar (failure was FSM-advance, not schema compile). All probes read-only; no config/code changed yet.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "colleague-320-tool-calls", "relates_to": "colleague-issue-320", "date": "2026-07-14", "box": "spark"}, "created": "2026-07-14T17:27:39.211406+00:00", "last_recall": "2026-07-31T13:22:52.104978+00:00", "recall_count": 2.0, "links": ["thor-serves-thor-lobe-2026-07-14", "unified-memory-budget-is-measured-truth-2026-07-14"], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "thor-serves-thor-lobe-2026-07-14", "hash": "69e7a5f411a88226fa2172e160bd60b96b470c5cd367d3e77977ee682195d73f", "content": "Deployment change (2026-07-14, PR #121 live test, operator-directed, permanent): the Jetson AGX Thor box now serves the thor-lobe shape \u2014 Gemma senses at max_model_len=131072 / gpu_mem_util=0.30 plus embedder+reranker; the Qwen cortex is DROPPED here (PRIMARY_FEASIBLE=false) with the honest referral PRIMARY_PEER_ORIGIN=http://spark.tail0be7e0.ts.net:8001 (the Spark hosts cortex; its lobes gateway is on port 8001). Callers on the Thor asking for model=main/hard/cortex get 404 role_infeasible with hosted_by \u2014 dial the Spark directly, nothing proxies. Gateway pinned to TestPyPI 0.43.0.dev239 until PR #121 releases 0.43.0 \u2014 re-pin after merge. Previous machine-as-brain deployment preserved at ~/.lobes.pre-accept-20260714T151429Z. The re-scaffold fixed the stale rerank-lane hang (#105/#119): rerank ordering passes in ~1.4s. Full PASS transcript: docs/evidence/2026-07-14-accept-thor-lobe-live-referral-thor.txt. Boot caveat stands: after teardown run sync + drop_caches before fleet up (unified-memory race; 16 restarts without it).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "decision", "record_metadata": {"issue": "#112", "pr": "#121", "box": "thor"}, "created": "2026-07-14T15:52:50.642616+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 5.5, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "colleague-320-tool-call-mangle-root-cause", "hash": "1592fa70f1407d998bd87856f4d54472f7779065b4b5a8e75fc44b8d187de8d1", "content": "Colleague #320 (cortex tool calls arrive mangled: name read_file\" + empty arguments, every work item delivers nothing) ROOT-CAUSED 2026-07-14 on the Spark GB10, hypothesis (a) CONFIRMED server-side: a two-way logging proxy captured the RAW gateway/vLLM response during a real colleague work run and the server itself already returns tool_call name=read_file\\\" args={} \u2014 colleague parsing is exonerated. Deterministic minimal repro (temp 0.0): colleague 4K-char system prompt + 14 tools + tool_choice=auto \u2192 the Qwen3.6-27B-MTP cortex drifts off its own chat template and emits a hallucinated \\n\\n[tool_id]{json} format (NOT the qwen3_coder XML its template prescribes); vLLM qwen3 parser salvage mangles it. The long-prompt theory was WRONG \u2014 4K chars suffices; nothing in colleague prompt teaches JSON tool format. FIX PATH: (1) strict:true on function tools arms xgrammar structural-tag constrained decoding (VLLM_ENFORCE_STRICT_TOOL_CALLING defaults True, but with tool_choice=auto it only arms if some tool declares strict:true \u2014 structural_tag_registry.get_model_structural_tag gate); (2) BUT the served vLLM build (0.23.1rc1.dev672+g93d8f834d) hardcodes reasoning=False at the call site (parser/abstract_parser.py _apply_structural_tag \u2192 get_structural_tag(request, reasoning=False)), so with thinking on, the grammar cannot accept the special token (id 248069) \u2192 FSM failure \u2192 HTTP 500 (grammar rejected tokens [,\\n\\n,,\\n]); (3) strict + chat_template_kwargs enable_thinking:false returns a CLEAN structured call (read_file {path: calc.py}) \u2014 proven live, MTP spec-decode active (77.8% draft acceptance) so constrained decoding + MTP coexist; (4) reasoning=True IS supported by this build xgrammar builtin (sequence: any_text \u2192 \u2192 \\n\\n \u2192 triggered tags) \u2014 only the call site never passes it. Candidate lobes fix: tiny --tool-parser-plugin subclassing Qwen3EngineToolParser overriding get_structural_tag to pass reasoning=True when the request has thinking enabled (ToolParserManager.import_tool_parser exists in the image); then colleague sends strict:true (or lobes gateway injects it, config-gated). All 14 colleague tool schemas compiled fine under xgrammar (failure was FSM-advance, not schema compile). All probes read-only; no config/code changed yet.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "colleague-320-tool-calls", "relates_to": "colleague-issue-320", "date": "2026-07-14", "box": "spark"}, "created": "2026-07-14T17:27:39.211406+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 2.5, "links": ["thor-serves-thor-lobe-2026-07-14", "unified-memory-budget-is-measured-truth-2026-07-14"], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "lobes-fleet-up-drops-audio-override", "hash": "092dcde3ef3b5c56392d6d8a5275614da30967578e07ed936a647de645076434", "content": "lobes fleet up resolves exactly -f docker-compose.yml -f docker-compose.audio.yml -f docker-compose.shape.yml (verified via lobes.runtime._compose._compose_files). It does NOT pick up a hand-written docker-compose.audio.override.yml. On the Spark box that override is what publishes the stt container on 127.0.0.1:9002, which reachy-mini-cli's default REACHY_STT_URL depends on -- so a lobes-driven fleet up would silently recreate stt WITHOUT the loopback publish and break reachy. Bring the audio overlay up with an explicit docker compose -f ... chain including the override until this is fixed.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "note", "record_metadata": {"source": "session-2026-07-17", "area": "runtime/_compose", "repo": "lobes-cli"}, "created": "2026-07-17T04:26:47.475910+00:00", "last_recall": "2026-07-17T05:22:37.990040+00:00", "recall_count": 1, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "eidetic-embed-is-hash-fallback-not-semantic", "hash": "b7d4b3fc0a93504696691b8b209f5a9a91bdf6dcb84a2a5eeacea7174e3c451c", "content": "FINDING (2026-07-20, spark-f8a9, verified by reading the installed package): eidetic's semantic recall on this box is NOT semantic \u2014 it has never reached a real embedding model. Installed eidetic-cli 0.10.1's memory/embed.py has _DEFAULT_BASE_URL='http://localhost:8101/v1' and _DEFAULT_MODEL='text-embedding-3-small' (NOT the 'localhost:8002 + Qwen3-Embedding-0.6B' the recall SKILL.md claims \u2014 that doc is STALE). Ports 8101 and 8002 are both CLOSED; the only open port is the lobes gateway on 8001. So embed_detect() throws on connect, catches, and returns (_local_embed(...), online=False) \u2014 a 128-dim deterministic lexical HASH (_EMBED_DIM=128, _local_embed at embed.py:63). Every approximate/hybrid recall runs on that hash. It is at least HONEST about it: embed_detect returns the `online` flag and hybrid refuses to fuse fallback cosine, so hybrid silently degrades to keyword-only rather than lying. HARD BLOCKER for wiring it up: _remote_embed builds its urllib Request with headers={'Content-Type': 'application/json'} ONLY \u2014 there is NO Authorization/API-key support anywhere in the embed client. The lobes gateway has inbound auth armed (GATEWAY_API_KEY; /v1/models returns invalid_api_key), so even setting EIDETIC_EMBED_URL=http://localhost:8001/v1 would 401 -> exception -> silent hash fallback. Fixing this is an UPSTREAM eidetic change (add API-key support to the embed client), not a lobes change. Also note the dim change 128 -> 1024 (0.6B) or 2560 (4B) invalidates every stored vector: adopting a real embedder means a full re-embed of the store either way.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "eidetic-embed-wiring", "host": "spark-f8a9", "date": "2026-07-20", "repo": "lobes-cli"}, "created": "2026-07-20T17:07:25.072568+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "realtime-pr2-scope-149", "hash": "fc0b14a5cdef96ca5096f4adef503b83c4ef6749c3b50a73edd559734e37fe9f", "content": "Scoping issue #149 (/v1/realtime WS, PR2) surfaced three facts that outlive the task. (1) The stdlib gateway CANNOT pass a WebSocket today: _HOP_BY_HOP strips Upgrade, do_GET has no tunnel, relay is buffered/SSE-rechunked \u2014 criterion-1 reachability is either new 101+bidirectional-socket-relay machinery in server.py or a published realtime container port. (2) PR1 already landed PR2 config: lobes/realtime/_settings.py ships the full VAD block (vad_threshold/vad_silence_ms/vad_prefix_padding_ms/default_turn_detection=server_vad/default_aec_mode=none/barge_in_*) BUT env.audio.example and docker-compose.audio.yml pass NONE of those keys, so container settings silently pin to defaults \u2014 the passthrough is real work. (3) Wire-format conflict: protocol.py pins CLIENT_SAMPLE_RATE=24000 (OpenAI Realtime PCM16, test-asserted) while #149 asks to stream at the 16kHz/512-sample VAD framing \u2014 a decision, not a given. Frame: .devague realtime-ws-server-vad-149 (c1-c17, q1-q3, s1-s12).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "scope-survey", "record_metadata": {"repo": "lobes-cli", "relates_to": "issue-149", "area": "realtime/gateway", "date": "2026-07-21"}, "created": "2026-07-21T09:23:55.158395+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} @@ -47,17 +47,18 @@ {"id": "fakes-must-model-blocking-rule", "hash": "587f16c9f32a85c661fd3194e725acb06e952b68d6df1dc5f7c7ee16395a3e50", "content": "STANDING RULE learned the hard way on lobes-cli PR #150 (#149 realtime WS): A TEST CANNOT VALIDATE A BEHAVIOUR THE TEST ITSELF DEFINES. Three separate defects in ONE PR were hidden by fake objects that did not model the real thing, each passing a green suite of 2200+ tests: (1) a fake reader whose read(n) returned data[:n] immediately, vs io.BufferedReader which on a blocking socket fills n bytes or hits EOF -> gateway read_head() hung on EVERY real connection; fix read1() + real socket.socketpair() tests. (2) a unit test asserting `leftover` was sent to `upstream` -- the WRONG DIRECTION enshrined as the contract -> every WS session died at open (RFC 6455 5.1: bridge must close on an unmasked server frame); only a live run could catch a direction the test picked. (3) _FakeSock.recv returning b\"\" as soon as its script ran dry, vs a real IDLE peer that just sits there -> a dead upstream stranded one gateway handler thread per open session, because shutdown(SHUT_WR) on the opposite socket does NOT wake a recv blocked on that same socket; fix: shutdown(SHUT_RDWR) on peer when either pump exits, and a _BlockingSock fake that waits on threading.Event. PRACTICE: (a) when a fake stands in for blocking I/O, model the BLOCKING contract or use a real socketpair; (b) when you assert a DIRECTION/ORDERING you chose, only an integration/live run validates it; (c) always verify a new regression test FAILS against the reintroduced bug before trusting it. Qodo independently found (2) and (3) \u2014 the diverse-review reflex plus live testing both earned their keep; neither alone was sufficient.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "lesson", "record_metadata": {"repo": "lobes-cli", "relates_to": "issue-149, pr-150", "area": "testing", "date": "2026-07-21"}, "created": "2026-07-21T14:03:39.021488+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "silent-failure-antipattern-voice-tools", "hash": "337c99c77ec7beb7632d5011ecb3ad34a21f4b8a6fdb8540f723442e8d500072", "content": "PATTERN learned across lobes-cli PR #150 + #152 (realtime voice work, 2026-07-21): IN AN INTERACTIVE/VOICE TOOL, EVERY SILENT FAILURE PATH MUST BE MADE TO SPEAK. Six separate defects in the voice loop all presented identically \u2014 the loop sat quietly until its idle timeout and printed \"nothing heard \u2014 ending\", indistinguishable from a user who simply did not speak: (1) the event reader caught EVERY read_frame exception and retried, so an EOF after disconnect spun forever; fix = socket.timeout continues, anything else stops the session and names it; (2) speak() ran the audio player with no timeout and paplay REALLY DID hang on a sink whose ALSA device reachy-mini-dae held exclusively \u2014 with the mic muted for the speak window this deafens the session permanently; fix = bounded per-backend timeout, fall through to next; (3) a mid-session mic EOF broke the feeder without setting stop; (4) arecord stderr was never piped so the no-audio message could not quote the ALSA error (the `if mic.stderr else \"\"` guard made it dead code); (5) a busy capture device looked exactly like a silent room; (6) terminate() left arecord orphaned holding the mic so the NEXT run failed busy. WHY IT MATTERS: the natural human response to silence is to TALK LOUDER, not to read logs \u2014 so a silent failure in a voice tool actively misleads the operator. ALSO: a /v1/realtime client MUST answer WebSocket PING with PONG \u2014 uvicorn pings ~every 20s and closes a peer that never pongs; both ends then report misleadingly (bridge logs a clean client_disconnect, client gets BrokenPipeError) so it looks like a network fault. A one-shot smoke run finishes inside one ping interval and never sees it. Isolate gateway-vs-client by bypassing the gateway and reproducing.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "lesson", "record_metadata": {"repo": "lobes-cli", "relates_to": "pr-150, pr-152, issue-151", "area": "testing/interactive-tools", "date": "2026-07-21"}, "created": "2026-07-21T15:45:42.604858+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "pr-90-fleet-reachability-truth-review-reflex", "hash": "91a2aafd09c1fb6859b760c68c8417e2866db446a86a7edc7c3c87dadb80c26c", "content": "PR #90 (agentculture/lobes-cli, 0.38.0, 2026-07-04) shipped fleet reachability truth as ONE PR fixing #84/#87/#89: (#84) lobes status fleet-aware via _compose.is_fleet() Dockerfile.gateway marker, single-model output byte-identical; (#87) GET /capabilities + every role endpoint advertises a client-reachable origin from the request Host header with a GATEWAY_PUBLIC_URL override for tunnels; (#89) stt/tts ready is a LIVE probe of the realtime bridge GET /v1/health/ready aggregate, gateway returns 503-warming vs 502-unreachable for /v1/audio/*, Chatterbox reports honest 503 cuda_context_poisoned. Built via the devague pipeline (/think spec -> /spec-to-plan 6 tasks/3 waves decomposed by FILE for disjointness -> /assign-to-workforce) with colleague local-vLLM as the wave-0 workforce and opus for the gateway integration. LESSON \u2014 the diverse-review reflex earns its keep: the ask-colleague review AND the Qodo agentic review each independently caught real semantic bugs in the SAME audio loaded/ready area. colleague caught loaded/ready conflation (a warming backend read as not-deployed). Qodo caught two more: probe_audio_ready caught only OSError so a malformed AUDIO_URL (non-numeric port -> urlsplit(...).port raises ValueError) crashed the handler uncaught; and build_role_registry did not clamp ready on audio_configured, so a caller passing audio_ready=True with AUDIO_URL unset got ready=True/loaded=False/endpoint=empty. Fix patterns: catch ValueError in URL-parsing probes exactly like open_upstream does; make public builders self-enforce the invariants their own docstrings promise. Run BOTH review passes on a committed diff before merge.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "pr-90-fleet-reachability", "relates_to": "issues-84-87-89", "date": "2026-07-04"}, "created": "2026-07-04T04:03:43.104710+00:00", "last_recall": "2026-07-24T18:41:34.170386+00:00", "recall_count": 5, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "spark-serves-spark-lobe-no-audio-2026-07-14", "hash": "70f2bfc02014f3787e6a39d7ada9548217fdb1ee0d8066b13a74b12f95f4c5d2", "content": "Deployment change (2026-07-14, operator-directed): the Spark GB10 box moves to the spark-lobe shape at lobes-cli 0.44.0 \u2014 cortex (Qwen3.6-27B-MTP) at full native 262144 / util 0.44, senses DROPPED with honest referral MULTIMODAL_PEER_ORIGIN=http://thor.tail0be7e0.ts.net:8000 (Thor gateway port is 8000, not 8001), embedder+reranker co-resident, GATEWAY_FORCE_STRICT_TOOLS=1 armed (colleague#320 fix, qwen3_coder_thinking plugin). The realtime AUDIO OVERLAY IS DROPPED on this box (operator: \"we can give up realtime for this machine\") \u2014 stt/tts stay in the six-role contract as feasible:true/ready:false. OPERATOR DIRECTION captured: ALL dropped/undeployed models must stay DECLARED as sleeping lobes (never erased from the contract) in preparation for a future fallback feature that serves a sleeping role from another machine \u2014 the referral half exists today (hosted_by), the proxy/fallback half is issue #115 (proxy-lobes follow-up). Env carries: CULTURE_VLLM_API_KEY, pressure thresholds=100; TTS_SPEED/OPENAI_API_KEY dropped with the overlay. Previous machine-as-brain deployment: ~/.lobes.pre-accept-20260714T183827Z (restore via accept-shape.sh --restore). Gotcha: two acceptance attempts failed on PyPI CDN propagation lag right after the 0.44.0 publish (pip in the image build could not see the fresh wheel; retry minutes later succeeds).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "decision", "record_metadata": {"box": "spark", "topic": "spark-lobe-golive", "relates_to": "issue-115, colleague-320", "date": "2026-07-14"}, "created": "2026-07-14T19:10:03.888285+00:00", "last_recall": "2026-07-31T13:22:52.104978+00:00", "recall_count": 10.0, "links": ["thor-serves-thor-lobe-2026-07-14", "colleague-320-tool-call-mangle-root-cause", "unified-memory-budget-is-measured-truth-2026-07-14"], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "issue-127-proxy-lobes-pairwise-auth-specced", "hash": "9defa4bc3dc4492ddf198c25e0dcefdbcbb5eb5656fecb33f8d9cf9103511272", "content": "Issue #127 (heterogeneous multi-node routing) scoped+specced+planned 2026-07-16 via full devague chain -> docs/specs/2026-07-16-proxy-lobes-pairwise-auth.md + docs/plans/2026-07-16-proxy-lobes-pairwise-auth.md (branch feat/127-proxy-lobes-pairwise-auth). USER DECISIONS: (q1) proxy opt-in is an EXPLICIT per-role knob _PEER_PROXY=true next to _PEER_ORIGIN \u2014 origin alone stays referral-only, live Spark/Thor referral deployments unchanged; (q2) inbound gateway auth = new GATEWAY_API_KEY env with CULTURE_VLLM_API_KEY fallback (both unset = no auth); (q3) delivery = #127 phase 1 only (proxy data plane + pairwise auth + existing role vocabulary + explainable surfaces), fanout/trace/runtime-aware routing parked as follow-ups. KEY DESIGN: pairwise keys never propagate (caller key authenticates only to the box dialed; proxy branch REPLACES Authorization with _PEER_API_KEY = the peer own inbound key); single-hop loop guard via X-Lobes-Proxied marker family; proxied responses carry X-Lobes-Proxied-By; peer down => 503 Retry-After; peer role_infeasible => honest relay, never chain; /v1/models lists a proxied role only when a peer-readiness probe (peer /v1/models with peer key) passes; capabilities show proxied:true + hosted_by, feasible stays false. GOTCHA FOUND: fleet gateway had NO inbound auth at all (docs/gateway-fleet.md:376 known limitation) \u2014 CULTURE_VLLM_API_KEY only ever protected the legacy single-model path.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "decision", "record_metadata": {"source": "claude-code-session", "topic": "proxy-lobes", "relates_to": "issue-127, issue-115, issue-112", "date": "2026-07-16"}, "created": "2026-07-16T12:43:07.190007+00:00", "last_recall": "2026-07-31T13:22:52.104978+00:00", "recall_count": 4.5, "links": ["spark-serves-spark-lobe-no-audio-2026-07-14", "issue-112-one-lobe-per-box-specced"], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "muse-role-gemma31b-design-2026-07-17", "hash": "e83179b7098e2c15e61485f58cb9f5c706ad94ef0eff525055a74b26ab3e1e78", "content": "muse role landed (PR #134, 0.46.0, 2026-07-17): the SEVENTH Colleague role \u2014 creative/ideation lobe, nvidia/Gemma-4-31B-IT-NVFP4 (NVIDIA modelopt NVFP4 \u2014 NOT compressed-tensors like the community 12B; 30.4 GiB weights; 256K native; plain-gemma4 line Gemma4ForConditionalGeneration with vision+audio configs; calibrated FP8 KV scales). KEY DESIGN: muse is the first OPT-IN CORE ROLE (shapes.OPT_IN_CORE_ROLES) \u2014 full Profile knobs (MUSE_* prefix, schema now 5 core roles) but machine-as-brain NEVER hosts it (31B cannot co-reside with cortex+senses on 128GB). DEFAULT_HOSTED_ROLES (6) = machine-as-brain identity set; COLLEAGUE_ROLES (7) = contract set. Byte-identity preserved: non-hosted opt-in core role renders NOTHING; gateway OPT_IN_BACKENDS rule makes unwired muse infeasible-by-default so model=muse on stale .env 404s role_infeasible instead of upward-fallback to cortex (the #92 trap). thor-muse shape carries the FULL muse declaration in [overrides.muse] (model + util 0.40 HYPOTHESIS + 131072 + modelopt + TRITON_ATTN); cards stay silent, base.toml vetoes. vllm-muse service is compose-profile-gated (COMPOSE_PROFILES=muse + MUSE_BASE_URL = the activation pair a hosting shape renders). MTP declared via google/gemma-4-31B-it-assistant (gemma4_assistant family) \u2014 UNMEASURED, first acceptance run gates it. Capability order: minor < multimodal < muse < primary (TIER_ROLE last-occurrence: muse inserted between senses and cortex). Deployment intent: Thor moves thor-lobe -> thor-muse; senses served by the ORIN (already live at orin.tail0be7e0.ts.net:8000 serving the 12B unauthenticated) with MULTIMODAL_PEER_ORIGIN + MULTIMODAL_PEER_PROXY=true on Thor. Everything muse is DECLARED/UNVALIDATED until accept-shape.sh thor-muse runs on the physical Thor (#108).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "decision", "record_metadata": {"topic": "muse-role", "relates_to": "PR-134, issue-81, issue-112", "date": "2026-07-17"}, "created": "2026-07-16T23:14:05.347710+00:00", "last_recall": "2026-07-31T13:22:52.104978+00:00", "recall_count": 3.0, "links": ["issue-112-one-lobe-per-box-specced", "issue-127-proxy-lobes-pairwise-auth-specced"], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "spark-serves-spark-lobe-no-audio-2026-07-14", "hash": "70f2bfc02014f3787e6a39d7ada9548217fdb1ee0d8066b13a74b12f95f4c5d2", "content": "Deployment change (2026-07-14, operator-directed): the Spark GB10 box moves to the spark-lobe shape at lobes-cli 0.44.0 \u2014 cortex (Qwen3.6-27B-MTP) at full native 262144 / util 0.44, senses DROPPED with honest referral MULTIMODAL_PEER_ORIGIN=http://thor.tail0be7e0.ts.net:8000 (Thor gateway port is 8000, not 8001), embedder+reranker co-resident, GATEWAY_FORCE_STRICT_TOOLS=1 armed (colleague#320 fix, qwen3_coder_thinking plugin). The realtime AUDIO OVERLAY IS DROPPED on this box (operator: \"we can give up realtime for this machine\") \u2014 stt/tts stay in the six-role contract as feasible:true/ready:false. OPERATOR DIRECTION captured: ALL dropped/undeployed models must stay DECLARED as sleeping lobes (never erased from the contract) in preparation for a future fallback feature that serves a sleeping role from another machine \u2014 the referral half exists today (hosted_by), the proxy/fallback half is issue #115 (proxy-lobes follow-up). Env carries: CULTURE_VLLM_API_KEY, pressure thresholds=100; TTS_SPEED/OPENAI_API_KEY dropped with the overlay. Previous machine-as-brain deployment: ~/.lobes.pre-accept-20260714T183827Z (restore via accept-shape.sh --restore). Gotcha: two acceptance attempts failed on PyPI CDN propagation lag right after the 0.44.0 publish (pip in the image build could not see the fresh wheel; retry minutes later succeeds).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "decision", "record_metadata": {"box": "spark", "topic": "spark-lobe-golive", "relates_to": "issue-115, colleague-320", "date": "2026-07-14"}, "created": "2026-07-14T19:10:03.888285+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 11.0, "links": ["thor-serves-thor-lobe-2026-07-14", "colleague-320-tool-call-mangle-root-cause", "unified-memory-budget-is-measured-truth-2026-07-14"], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "issue-127-proxy-lobes-pairwise-auth-specced", "hash": "9defa4bc3dc4492ddf198c25e0dcefdbcbb5eb5656fecb33f8d9cf9103511272", "content": "Issue #127 (heterogeneous multi-node routing) scoped+specced+planned 2026-07-16 via full devague chain -> docs/specs/2026-07-16-proxy-lobes-pairwise-auth.md + docs/plans/2026-07-16-proxy-lobes-pairwise-auth.md (branch feat/127-proxy-lobes-pairwise-auth). USER DECISIONS: (q1) proxy opt-in is an EXPLICIT per-role knob _PEER_PROXY=true next to _PEER_ORIGIN \u2014 origin alone stays referral-only, live Spark/Thor referral deployments unchanged; (q2) inbound gateway auth = new GATEWAY_API_KEY env with CULTURE_VLLM_API_KEY fallback (both unset = no auth); (q3) delivery = #127 phase 1 only (proxy data plane + pairwise auth + existing role vocabulary + explainable surfaces), fanout/trace/runtime-aware routing parked as follow-ups. KEY DESIGN: pairwise keys never propagate (caller key authenticates only to the box dialed; proxy branch REPLACES Authorization with _PEER_API_KEY = the peer own inbound key); single-hop loop guard via X-Lobes-Proxied marker family; proxied responses carry X-Lobes-Proxied-By; peer down => 503 Retry-After; peer role_infeasible => honest relay, never chain; /v1/models lists a proxied role only when a peer-readiness probe (peer /v1/models with peer key) passes; capabilities show proxied:true + hosted_by, feasible stays false. GOTCHA FOUND: fleet gateway had NO inbound auth at all (docs/gateway-fleet.md:376 known limitation) \u2014 CULTURE_VLLM_API_KEY only ever protected the legacy single-model path.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "decision", "record_metadata": {"source": "claude-code-session", "topic": "proxy-lobes", "relates_to": "issue-127, issue-115, issue-112", "date": "2026-07-16"}, "created": "2026-07-16T12:43:07.190007+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 5.5, "links": ["spark-serves-spark-lobe-no-audio-2026-07-14", "issue-112-one-lobe-per-box-specced"], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "muse-role-gemma31b-design-2026-07-17", "hash": "e83179b7098e2c15e61485f58cb9f5c706ad94ef0eff525055a74b26ab3e1e78", "content": "muse role landed (PR #134, 0.46.0, 2026-07-17): the SEVENTH Colleague role \u2014 creative/ideation lobe, nvidia/Gemma-4-31B-IT-NVFP4 (NVIDIA modelopt NVFP4 \u2014 NOT compressed-tensors like the community 12B; 30.4 GiB weights; 256K native; plain-gemma4 line Gemma4ForConditionalGeneration with vision+audio configs; calibrated FP8 KV scales). KEY DESIGN: muse is the first OPT-IN CORE ROLE (shapes.OPT_IN_CORE_ROLES) \u2014 full Profile knobs (MUSE_* prefix, schema now 5 core roles) but machine-as-brain NEVER hosts it (31B cannot co-reside with cortex+senses on 128GB). DEFAULT_HOSTED_ROLES (6) = machine-as-brain identity set; COLLEAGUE_ROLES (7) = contract set. Byte-identity preserved: non-hosted opt-in core role renders NOTHING; gateway OPT_IN_BACKENDS rule makes unwired muse infeasible-by-default so model=muse on stale .env 404s role_infeasible instead of upward-fallback to cortex (the #92 trap). thor-muse shape carries the FULL muse declaration in [overrides.muse] (model + util 0.40 HYPOTHESIS + 131072 + modelopt + TRITON_ATTN); cards stay silent, base.toml vetoes. vllm-muse service is compose-profile-gated (COMPOSE_PROFILES=muse + MUSE_BASE_URL = the activation pair a hosting shape renders). MTP declared via google/gemma-4-31B-it-assistant (gemma4_assistant family) \u2014 UNMEASURED, first acceptance run gates it. Capability order: minor < multimodal < muse < primary (TIER_ROLE last-occurrence: muse inserted between senses and cortex). Deployment intent: Thor moves thor-lobe -> thor-muse; senses served by the ORIN (already live at orin.tail0be7e0.ts.net:8000 serving the 12B unauthenticated) with MULTIMODAL_PEER_ORIGIN + MULTIMODAL_PEER_PROXY=true on Thor. Everything muse is DECLARED/UNVALIDATED until accept-shape.sh thor-muse runs on the physical Thor (#108).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "decision", "record_metadata": {"topic": "muse-role", "relates_to": "PR-134, issue-81, issue-112", "date": "2026-07-17"}, "created": "2026-07-16T23:14:05.347710+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 4.0, "links": ["issue-112-one-lobe-per-box-specced", "issue-127-proxy-lobes-pairwise-auth-specced"], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "thor-muse-live-bringup-20260717", "hash": "c73f9a932dff115c5c6ae1746e085edcbae9424c5efafffb3c01920d1c340d5f", "content": "thor-muse live bring-up lessons (2026-07-17, PR #134): (1) muse budget MEASURED on the physical Thor \u2014 util 0.40 refused (0.6 GiB KV left after the 262144-token profiling pass), util 0.55 boots healthy (26.47 GiB KV pool / 611,415 fp8 tokens / 2.33x concurrency at the full 256K window). (2) compose depends_on service_healthy can ORPHAN a dependent: vllm-embed crashed once during the concurrent cold boot (Available KV cache memory: -3.83 GiB), self-healed via its restart policy, but compose never retro-starts the dependent \u2014 vllm-muse sat in created state (Pid 0, no logs, invisible in docker ps) until a manual `lobes up muse --apply`; with deps healthy the 31B loads in ~6.5 min. (3) Status discipline: the budget is measured but the thor-muse shape stays UNVALIDATED until scripts/accept-shape.sh passes and its transcript lands in docs/evidence/ (#108) \u2014 that acceptance run is the open follow-up.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "note", "record_metadata": {"source": "PR #134 live bring-up", "repo": "lobes-cli", "machine": "thor"}, "created": "2026-07-17T03:05:37.994416+00:00", "last_recall": "2026-07-31T13:22:44.488792+00:00", "recall_count": 1.0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "spark-proxy-golive-senses-orin-muse-thor-20260717", "hash": "fcea0b2d605daa116476a6ea9a918c8a83a7bd30c51a3ecc0ee9533596e8487d", "content": "Spark GB10 proxy wiring (2026-07-17, lobes 0.46.0): senses->orin and muse->thor are both live on the spark-lobe box. TWO TRAPS COST TIME. (1) COMPOSE PASSTHROUGH: a deployed docker-compose.yml scaffolded by an OLDER lobes has no MUSE_* passthrough in the gateway service, so MUSE_PEER_ORIGIN/PROXY in .env silently never reach the container and the proxy will NOT arm even with a 0.46.0 gateway image \u2014 /capabilities shows muse proxied=None. The deployed compose here was a 0.45.0 template HAND-PATCHED during the #127 proxy test (4 added lines: GATEWAY_API_KEY, CULTURE_VLLM_API_KEY, MULTIMODAL_PEER_PROXY, MULTIMODAL_PEER_API_KEY). Fix = copy the packaged FLEET template (templates/fleet/docker-compose.yml, NOT the top-level single-model one) over it; only 3 stale comment lines were lost. Bump MODEL_GEAR_VERSION too \u2014 it pins the gateway image's pip lobes-cli. (2) SHAPE-OVERRIDE TRAP: NEVER run bare `docker compose up -d --build gateway` in ~/.lobes \u2014 the gateway's base depends_on fans out to every core service, so without `-f docker-compose.shape.yml` the DROPPED vllm-multimodal boots and eats the reclaimed GPU budget, and cortex gets recreated (27B reload). Always pass -f docker-compose.yml -f docker-compose.shape.yml. `lobes fleet up --apply` is the intended path but is BROKEN on this box: it auto-includes docker-compose.audio.yml which references a missing Dockerfile.chatterbox. AUTH STATE: thor was re-scaffolded to thor-muse and now sets NO inbound key, so its old key mg-6kYbjF1... is STALE; both MUSE_PEER_API_KEY and MULTIMODAL_PEER_API_KEY are deliberately EMPTY (never send a key to a box with no gate). This Spark DOES gate inbound via CULTURE_VLLM_API_KEY \u2014 callers get 401 without it. VERIFIED: senses 200 + X-Lobes-Proxied-By orin (text 17+25=42, vision identified a circle); muse honest 503 backend_unavailable while thor's muse is down. STT (parakeet, model-gear-stt) works standalone on 127.0.0.1:9002 (perfect JFK transcription) but is NOT gateway-exposed \u2014 AUDIO_URL is empty so /v1/audio/* 404s 'audio endpoints are not configured'; the gateway fans audio to the realtime bridge, which this box never built. UPDATE (same day, ~06:40 local): thor muse FINISHED BOOTING and muse->thor now serves 200 OK end-to-end (model nvidia/Gemma-4-31B-IT-NVFP4, X-Lobes-Proxied-By thor, vivid creative reply); /capabilities muse ready=true proxied=true, /v1/models lists both proxied peers. FIRST LIVE BOOT of the muse role \u2014 but it did NOT validate the shipped thor-muse.toml: the live Thor runs HAND-TUNED MUSE_GPU_MEM_UTIL=0.55 / MUSE_MAX_MODEL_LEN=262144, whereas the built-in shape ships the UNPROVEN hypothesis 0.40 / 131072. So lobes init --shape thor-muse renders values that have never booted, while the values that did boot are not in the shape. Reconcile before any doc/support-table claims thor-muse validated (#108 rule). Thor muse boot profile: engine init 200.51s (compilation 79.53s), ~95/122 GB system mem used, MTP draft google/gemma-4-31B-it-assistant armed via --speculative-config. Three-box mesh brain now whole: Spark cortex + Orin senses + Thor muse, one endpoint.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "fact", "record_metadata": {"box": "spark", "topic": "proxy-lobes-golive", "relates_to": "issue-115, issue-127, PR-134", "date": "2026-07-17"}, "created": "2026-07-17T03:45:05.791703+00:00", "last_recall": "2026-07-31T13:22:44.488792+00:00", "recall_count": 3.0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "spark-proxy-golive-senses-orin-muse-thor-20260717", "hash": "fcea0b2d605daa116476a6ea9a918c8a83a7bd30c51a3ecc0ee9533596e8487d", "content": "Spark GB10 proxy wiring (2026-07-17, lobes 0.46.0): senses->orin and muse->thor are both live on the spark-lobe box. TWO TRAPS COST TIME. (1) COMPOSE PASSTHROUGH: a deployed docker-compose.yml scaffolded by an OLDER lobes has no MUSE_* passthrough in the gateway service, so MUSE_PEER_ORIGIN/PROXY in .env silently never reach the container and the proxy will NOT arm even with a 0.46.0 gateway image \u2014 /capabilities shows muse proxied=None. The deployed compose here was a 0.45.0 template HAND-PATCHED during the #127 proxy test (4 added lines: GATEWAY_API_KEY, CULTURE_VLLM_API_KEY, MULTIMODAL_PEER_PROXY, MULTIMODAL_PEER_API_KEY). Fix = copy the packaged FLEET template (templates/fleet/docker-compose.yml, NOT the top-level single-model one) over it; only 3 stale comment lines were lost. Bump MODEL_GEAR_VERSION too \u2014 it pins the gateway image's pip lobes-cli. (2) SHAPE-OVERRIDE TRAP: NEVER run bare `docker compose up -d --build gateway` in ~/.lobes \u2014 the gateway's base depends_on fans out to every core service, so without `-f docker-compose.shape.yml` the DROPPED vllm-multimodal boots and eats the reclaimed GPU budget, and cortex gets recreated (27B reload). Always pass -f docker-compose.yml -f docker-compose.shape.yml. `lobes fleet up --apply` is the intended path but is BROKEN on this box: it auto-includes docker-compose.audio.yml which references a missing Dockerfile.chatterbox. AUTH STATE: thor was re-scaffolded to thor-muse and now sets NO inbound key, so its old key mg-6kYbjF1... is STALE; both MUSE_PEER_API_KEY and MULTIMODAL_PEER_API_KEY are deliberately EMPTY (never send a key to a box with no gate). This Spark DOES gate inbound via CULTURE_VLLM_API_KEY \u2014 callers get 401 without it. VERIFIED: senses 200 + X-Lobes-Proxied-By orin (text 17+25=42, vision identified a circle); muse honest 503 backend_unavailable while thor's muse is down. STT (parakeet, model-gear-stt) works standalone on 127.0.0.1:9002 (perfect JFK transcription) but is NOT gateway-exposed \u2014 AUDIO_URL is empty so /v1/audio/* 404s 'audio endpoints are not configured'; the gateway fans audio to the realtime bridge, which this box never built. UPDATE (same day, ~06:40 local): thor muse FINISHED BOOTING and muse->thor now serves 200 OK end-to-end (model nvidia/Gemma-4-31B-IT-NVFP4, X-Lobes-Proxied-By thor, vivid creative reply); /capabilities muse ready=true proxied=true, /v1/models lists both proxied peers. FIRST LIVE BOOT of the muse role \u2014 but it did NOT validate the shipped thor-muse.toml: the live Thor runs HAND-TUNED MUSE_GPU_MEM_UTIL=0.55 / MUSE_MAX_MODEL_LEN=262144, whereas the built-in shape ships the UNPROVEN hypothesis 0.40 / 131072. So lobes init --shape thor-muse renders values that have never booted, while the values that did boot are not in the shape. Reconcile before any doc/support-table claims thor-muse validated (#108 rule). Thor muse boot profile: engine init 200.51s (compilation 79.53s), ~95/122 GB system mem used, MTP draft google/gemma-4-31B-it-assistant armed via --speculative-config. Three-box mesh brain now whole: Spark cortex + Orin senses + Thor muse, one endpoint.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "fact", "record_metadata": {"box": "spark", "topic": "proxy-lobes-golive", "relates_to": "issue-115, issue-127, PR-134", "date": "2026-07-17"}, "created": "2026-07-17T03:45:05.791703+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 4.0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "lobes-init-apply-is-not-a-repair-path", "hash": "d4e7cf97c5624eae62ba0d2bede92a7db832086d60703a72a770776bace9481e", "content": "lobes init --audio --apply is NOT a repair path for a partially-scaffolded deployment. write_scaffold (lobes/runtime/_compose.py:322) refuses when ANY target file already exists ('re-run with --force'), and --force then overwrites EVERY template including .env -- which on a live box destroys the shape reclaim values, the *_PEER_* proxy config and the gateway API key. To heal a partial scaffold, hand-copy the missing files from the installed wheel's lobes/templates/fleet/ and append only the missing keys from env.audio.example. Skip HF_CACHE when appending: docker compose env_file semantics let the LAST duplicate key win, so appending a blank HF_CACHE= would clobber a real value. Found 2026-07-17 repairing the Spark audio overlay, where Dockerfile.chatterbox + Dockerfile.realtime and ALL audio .env keys were missing (append_audio_env had never run), which is why chatterbox/realtime never built and the gateway had AUDIO_URL empty.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "note", "record_metadata": {"source": "session-2026-07-17", "area": "init/scaffold", "repo": "lobes-cli"}, "created": "2026-07-17T04:26:47.471525+00:00", "last_recall": "2026-07-21T09:17:47.785433+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "lobes-audio-roles-outside-proxy-lobes", "hash": "02d81cd69c466b358de4fbbdd889108c534fbf1e3f41fabe64899f4bc3ca8a8d", "content": "The stt/tts roles are OUTSIDE the proxy-lobes / honest-referral machinery. PEER_ORIGIN_ENV, PEER_PROXY_ENV and PEER_API_KEY_ENV (lobes/gateway/_config.py) cover only primary/multimodal/muse/embed/rerank, and ROLE_BACKEND (lobes/roles.py:65) has no stt/tts entries; _audio_role() (roles.py:593) takes no peer argument, so audio roles can never carry hosted_by and annotate_peer_referrals skips them. There is no TTS_PEER_ORIGIN. The only cross-box lever is AUDIO_URL, and it breaks four proxy-lobes guarantees: (a) it is ONE url for the WHOLE /v1/audio/* namespace, so TTS-remote + STT-local is not expressible; (b) handle_audio_post uses filter_headers, which drops only hop-by-hop headers, so the caller's Authorization is forwarded verbatim to the peer -- proxy-lobes deliberately strips it; (c) no X-Lobes-Proxied loop guard / 508 and no X-Lobes-Proxied-By attribution; (d) capabilities would report tts endpoint= with no hosted_by, i.e. claim the local box hosts TTS. Extending referral/proxy to the audio lane is unbuilt work.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "note", "record_metadata": {"source": "session-2026-07-17", "area": "gateway/proxy-lobes", "repo": "lobes-cli"}, "created": "2026-07-17T04:26:47.475135+00:00", "last_recall": "2026-07-31T13:22:44.488792+00:00", "recall_count": 5.0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "lobes-audio-roles-outside-proxy-lobes", "hash": "02d81cd69c466b358de4fbbdd889108c534fbf1e3f41fabe64899f4bc3ca8a8d", "content": "The stt/tts roles are OUTSIDE the proxy-lobes / honest-referral machinery. PEER_ORIGIN_ENV, PEER_PROXY_ENV and PEER_API_KEY_ENV (lobes/gateway/_config.py) cover only primary/multimodal/muse/embed/rerank, and ROLE_BACKEND (lobes/roles.py:65) has no stt/tts entries; _audio_role() (roles.py:593) takes no peer argument, so audio roles can never carry hosted_by and annotate_peer_referrals skips them. There is no TTS_PEER_ORIGIN. The only cross-box lever is AUDIO_URL, and it breaks four proxy-lobes guarantees: (a) it is ONE url for the WHOLE /v1/audio/* namespace, so TTS-remote + STT-local is not expressible; (b) handle_audio_post uses filter_headers, which drops only hop-by-hop headers, so the caller's Authorization is forwarded verbatim to the peer -- proxy-lobes deliberately strips it; (c) no X-Lobes-Proxied loop guard / 508 and no X-Lobes-Proxied-By attribution; (d) capabilities would report tts endpoint= with no hosted_by, i.e. claim the local box hosts TTS. Extending referral/proxy to the audio lane is unbuilt work.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "note", "record_metadata": {"source": "session-2026-07-17", "area": "gateway/proxy-lobes", "repo": "lobes-cli"}, "created": "2026-07-17T04:26:47.475135+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 6.0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "branch-prune-content-containment", "hash": "4384c183f81145334d023295d878a7823d9c76fd6052f3f4ae7af456249cee5c", "content": "Pruning stale branches in lobes-cli: squash merges make every commit-based safety check useless. `git branch -d`, `git cherry`, and merge-base all report already-landed work as unmerged, because the squash rewrites the patch id. The check that actually answers 'is this branch disposable?' is CONTENT CONTAINMENT: `comm -23 <(git ls-tree -r --name-only BRANCH | sort) <(git ls-tree -r --name-only main | sort)` \u2014 does the branch hold any FILE main lacks. A large 'insertions vs main' count is a red herring: stale branches routinely show 200-300 such lines that are only superseded drafts (e.g. constructors taking individual params before a config-dataclass refactor, older close(reason=...) signatures), while the huge DELETION counts are the real signal that main is far ahead. Two renamed paths in this repo make old branches look like they hold unique work when they do not: `lobes/profiles.py` (refactored into the `lobes/profiles/` package) and `lobes/vllm_plugins/_thinking.py` (folded into `qwen3_thinking_tool_parser.py` \u2014 its `effective_reasoning()` is present verbatim there). Verified 2026-07-22 across 61 branches: those two were the ONLY unique files repo-wide, so all 56 non-colleague branches were safe to delete (62 branches -> 2). Always record SHAs first with `git for-each-ref --format='%(refname:short) %(objectname) %(committerdate:short)' refs/heads/`. Gotcha: a bulk `git branch -D $(...)` using command substitution is refused by the permission classifier \u2014 pass explicit branch names instead.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "note", "record_metadata": {"source": "session", "repo": "lobes-cli", "topic": "git-hygiene", "verified": "2026-07-22"}, "created": "2026-07-21T23:49:08.449828+00:00", "last_recall": "2026-07-21T23:49:15.141038+00:00", "recall_count": 1, "links": ["colleague-clean-reap-semantics"], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "colleague-clean-reap-semantics", "hash": "e6aa7447ae24323df115e59bebfc0eaa4b43eead884bc3e46749c1e45765a5fb", "content": "`colleague/*` branches must be reaped with the `colleague clean` CLI, never `git branch -D` \u2014 the CLI also reaps orphaned .colleague/ artifacts, isolation worktrees, dead background runs and experiments that a branch delete would strand. Non-obvious reaping rule: bare `colleague clean` keeps every branch it considers healthy, and `--merged` only ever reaps the `...second-independent-mind...` flavour (the explore/write modes, which can produce mergeable work). The `...independent-reviewer...` branches are read-only throwaway review worktrees that are NEVER merged by design, so `--merged` will keep them forever \u2014 they need `--older-than DAYS`. Observed 2026-07-22: bare clean kept 5/5; `--merged` reaped exactly the 2 second-independent-mind branches; `--older-than 5` then reaped the 2 reviewer branches older than a week and kept the 2-day-old one. See [[branch-prune-content-containment]] for the safety check applied to non-colleague branches.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "note", "record_metadata": {"source": "session", "repo": "lobes-cli", "topic": "git-hygiene", "verified": "2026-07-22"}, "created": "2026-07-21T23:49:08.452385+00:00", "last_recall": "2026-07-21T23:49:15.141038+00:00", "recall_count": 1, "links": ["branch-prune-content-containment"], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "stt-cpu-fallback-vs-cuda-gate-155", "hash": "c9abfe886857fdcfe1362e26ab724e59b039332f309f78eafdf1b53228f9a79a", "content": "Issue #155 (lobes-cli) central unknown SETTLED EMPIRICALLY on host spark 2026-07-24: the Parakeet stt gear was serving CORRECT transcripts on CPU while its CUDA readiness gate 503'd \u2014 reading 1, not reading 2 (not hollow 200s). METHOD worth reusing: the reporter was blocked because they had no speech sample and chatterbox (the in-fleet TTS) was itself down. The stt container's OWN base image ships known-content speech at /usr/share/sounds/alsa/*.wav \u2014 Front_Center.wav literally says 'front center'. docker cp it out, POST to 127.0.0.1:9002/v1/audio/transcriptions, got {\"text\":\"Front, center.\"} in 1.68s. Cross-check: nvidia-smi --query-compute-apps showed NO stt process holding GPU memory (only 4 VLLM EngineCore + reachy-mini-cli), so the model was demonstrably on CPU. ~1.1x realtime latency \u2014 fine for batch transcription, likely too slow for the /v1/realtime server_vad session, so 'degraded' may need different routing per lane. ROOT CAUSE of the GPU loss is NOT lobes code: model-gear-stt and model-gear-chatterbox (both 5h old) report 'Failed to initialize NVML: Unknown Error', torch.cuda.device_count()==0, CUDA_VISIBLE_DEVICES unset \u2014 DESPITE a valid docker DeviceRequest and NVIDIA_VISIBLE_DEVICES=all. A FRESH `docker run --rm --gpus all nvidia/cuda` container sees 'GPU 0: NVIDIA GB10' fine. So running containers lost their device cgroup mid-life (systemd 'Reloading' events in journalctl); vLLM containers survive because they already hold a CUDA context and never re-init. DIAGNOSTIC RULE: to tell 'container lost the GPU' from 'host GPU broken', always run the fresh-container test \u2014 host nvidia-smi alone does not distinguish them. WHY IT SURFACES AS A CONTRADICTION: listen_server.py get_model() calls ASRModel.from_pretrained() and never pins a device, so NeMo silently picks CPU at load time; the probe then tests torch.zeros(1, device='cuda'), which is a fact about the PROCESS, not about the loaded model's device. evaluate_readiness() in lobes/realtime/_readiness.py is two-state so CPU-serving has no representable value. Also confirmed: bare `python3` in the stt healthcheck is FINE (Dockerfile.parakeet base is scitrera/dgx-spark-vllm, which has /usr/bin/python3); the adjacent chatterbox comment warns about the DIFFERENT nvidia/cuda base, where python3 genuinely is missing. And GET /v1/realtime 404 was version skew \u2014 deployed gateway is lobes 0.52.3, the route shipped in 0.53.0; current code refuses a plain GET with 426 not_an_upgrade.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "investigation", "record_metadata": {"source": "claude-code-session", "topic": "stt-readiness-cpu-fallback", "relates_to": "issue-155", "host": "spark", "date": "2026-07-24"}, "created": "2026-07-24T18:47:32.086471+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "sonarcloud-key-and-sonar-sh-status-filter", "hash": "33667dbb66aa2ee46fefa5e889f5ff8eec5fe1ea96bd1c91feb7c886bc4c0dfd", "content": "lobes-cli SonarCloud gotchas (2026-07-24). (1) PROJECT KEY IS THE LEGACY PRE-RENAME NAME: sonar.projectKey=agentculture_model-gear (see sonar-project.properties), NOT agentculture_lobes-cli. SONAR_PROJECT is UNSET in the shell, so every sonarclaude call needs `export SONAR_PROJECT=agentculture_model-gear` or --project. Consequence: `agex pr read` / the cicd skill prints 'Project not on SonarCloud (skipped)' because agex derives the key from the repo name \u2014 that message is WRONG, Sonar does run on PRs (verified: PR #156 shows a passing 'SonarCloud Code Analysis' check linking to sonarcloud.io/dashboard?id=agentculture_model-gear&pullRequest=156). Don't conclude the project is unregistered from that line. (2) sonar.sh `issues` HAS NO STATUS FILTER \u2014 cmd_issues builds params as componentKeys+ps(+severities/types) and never passes resolved=false or statuses=, so /api/issues/search returns RESOLVED and CLOSED issues mixed with OPEN ones. This is actively misleading: the default listing showed a '[BLOCKER] VULNERABILITY pythonsecurity:S2083 lobes/runtime/_env.py:98' that is actually RESOLVED, and a CLOSED S5131. Always pull --raw and filter status in ('OPEN','CONFIRMED','REOPENED') client-side. Real counts on 2026-07-24: 17 returned = 10 OPEN + 4 RESOLVED + 3 CLOSED, and the 10 open contained NO blockers and NO vulnerabilities. Worth fixing in the vendored copy (owned by this repo per cite-don't-import) and filing upstream on steward. (3) Quality gate was PASSED with those 10 open \u2014 the gate scores NEW code only, so open issues in older code never fail it; check `issues` separately from `status`.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "sonarcloud-tooling", "repo": "lobes-cli", "date": "2026-07-24"}, "created": "2026-07-24T20:23:33.618027+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "multimodal-base-url-compose-default-traps-loaded", "hash": "64323c0f0c4ce28e587aee3b77ca1b6917b6eb6a54b0f5dc2a58524c2dc2993d", "content": "lobes fleet compose: MULTIMODAL_BASE_URL is the ONLY optional generate backend with a NON-EMPTY compose default (MULTIMODAL_BASE_URL=${MULTIMODAL_BASE_URL:-http://vllm-multimodal:8000}); MINOR_BASE_URL and MUSE_BASE_URL both default to empty. Because ${VAR:-default} substitutes the default for an UNSET *or EMPTY* value, you CANNOT unwire multimodal from a deployment .env at all \u2014 commenting the line out and setting it explicitly empty both yield the default (verified live on spark 2026-07-25: commented out, recreated gateway with -f docker-compose.yml -f docker-compose.shape.yml up -d --no-deps gateway, container env came back identical). Consequence: gateway/_config._optional_backend wires a multimodal Backend on EVERY fleet deployment, so roles.py:455 (loaded = backend is not None) reports senses loaded=true even on a box that has never run a vllm-multimodal container. That is why a proxied senses reports loaded=true while a proxied muse reports loaded=false \u2014 NOT stale per-box .env wiring, as first assumed. Fixed at the display layer in 0.54.2 (capabilities table renders by-proxy from feasible+proxied, ignoring loaded); the payload-level inaccuracy is issue #158. Dropping the default is a behaviour change \u2014 machine-as-brain boxes rely on it to wire senses.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "fact", "record_metadata": {"topic": "compose-defaults", "relates_to": "issue-158, PR-159", "box": "spark", "date": "2026-07-25"}, "created": "2026-07-25T09:00:09.357963+00:00", "last_recall": "2026-07-25T09:15:26.048389+00:00", "recall_count": 1, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "spark-compose-bringup-needs-all-four-f-files", "hash": "26be8f3cf414b379cfd2ab52c32982ea261ecbd7b22e2b1c2c75d4da4e1bd92c", "content": "Spark ~/.lobes bring-up MUST name ALL FOUR compose files, not two. The previously-recorded rule 'always pass -f docker-compose.yml -f docker-compose.shape.yml' is INCOMPLETE and actively caused a live defect. Passing ANY explicit -f suppresses docker compose's auto-load of docker-compose.override.yml. Correct invocation: docker compose -f docker-compose.yml -f docker-compose.audio.yml -f docker-compose.shape.yml -f docker-compose.override.yml. MEASURED CONSEQUENCE (2026-07-31): services had been brought up with inconsistent overlay sets (gateway from 2 files, vllm-embed-deep from 4), so the running gateway carried ZERO EMBED_DEEP_* env vars and Qwen/Qwen3-Embedding-4B was ABSENT from /v1/models -- the 4B embed gear ran Up 4 days healthy holding GPU budget while completely unroutable. Diagnose with: docker inspect --format '{{index .Config.Labels \"com.docker.compose.project.config_files\"}}' -- if two containers show different file lists, the stack is split-brain. Fixed by re-scaffolding docker-compose.yml from the packaged fleet template (which carries vllm-embed-deep + EMBED_DEEP_* natively) and trimming override.yml to ONLY the stt loopback :9002 publish that reachy-mini-cli needs. Also: the shape.yml !resets gateway depends_on, so `up -d gateway` alone does NOT fan out to dropped services -- rebuilding just the gateway is safe on this box.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"box": "spark", "topic": "compose-overlays", "repo": "lobes-cli", "date": "2026-07-31"}, "created": "2026-07-31T14:41:45.072853+00:00", "last_recall": null, "recall_count": 0, "links": ["spark-proxy-golive-senses-orin-muse-thor-20260717"], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "worker-peer-proxy-inert-before-0548", "hash": "97f2a557b8c41ec2a0da164668e7a5ab5d7d1ff9cf847308373d0f915f25003e", "content": "lobes worker peer-proxy was SILENTLY INERT from 0.54.6 (when the worker role landed) until the 0.54.8 fix. lobes/gateway/_config.py's PEER_PROXY_ENV carries eight roles; lobes/gateway/server.py's _PEER_SERVED_NAME_ENV and _PEER_ROLE_HINT carried only the five pre-worker ones. _peer_served_name therefore returned '' for worker EVEN WITH WORKER_SERVED_NAME set (that var was never looked up), so peer_specs_from_table dropped the role at its 'if not served_name: continue' guard -> no ReadinessCache peer probe, no /v1/models entry, no _proxied_owner match, so WORKER_PEER_PROXY=true forwarded nothing and fell through to the referral-only role_infeasible 404. It hit exactly the shape that needs it: worker is an opt-in core role, so only a worker-hosting shape renders WORKER_BASE_URL; a box that merely REACHES worker has no wired Backend to resolve off and lands on the two broken tables. CI COULD NOT SEE IT: every pre-existing worker proxy test used _worker_dropped_env(), which sets WORKER_BASE_URL -- a wired-but-infeasible shape that resolves via the wired backend and never touches either table; that helper's own docstring concluded 'no server.py change required', true only for the shape it tested. LESSON: when adding a role to PEER_PROXY_ENV, also teach BOTH server.py tables, and test the UNWIRED shape. Guard now exists: tests/test_gateway_proxy.py::test_every_proxyable_role_resolves_a_served_name iterates PEER_PROXY_ENV. Found by a devague falsification task whose whole job was to DISPROVE 'the proxy needs no lobes/ change'.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"topic": "proxy-lobes", "repo": "lobes-cli", "relates_to": "PR-165, issue-115, issue-127", "date": "2026-07-31"}, "created": "2026-07-31T14:41:45.075714+00:00", "last_recall": null, "recall_count": 0, "links": ["spark-compose-bringup-needs-all-four-f-files"], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "multimodal-base-url-compose-default-traps-loaded", "hash": "64323c0f0c4ce28e587aee3b77ca1b6917b6eb6a54b0f5dc2a58524c2dc2993d", "content": "lobes fleet compose: MULTIMODAL_BASE_URL is the ONLY optional generate backend with a NON-EMPTY compose default (MULTIMODAL_BASE_URL=${MULTIMODAL_BASE_URL:-http://vllm-multimodal:8000}); MINOR_BASE_URL and MUSE_BASE_URL both default to empty. Because ${VAR:-default} substitutes the default for an UNSET *or EMPTY* value, you CANNOT unwire multimodal from a deployment .env at all \u2014 commenting the line out and setting it explicitly empty both yield the default (verified live on spark 2026-07-25: commented out, recreated gateway with -f docker-compose.yml -f docker-compose.shape.yml up -d --no-deps gateway, container env came back identical). Consequence: gateway/_config._optional_backend wires a multimodal Backend on EVERY fleet deployment, so roles.py:455 (loaded = backend is not None) reports senses loaded=true even on a box that has never run a vllm-multimodal container. That is why a proxied senses reports loaded=true while a proxied muse reports loaded=false \u2014 NOT stale per-box .env wiring, as first assumed. Fixed at the display layer in 0.54.2 (capabilities table renders by-proxy from feasible+proxied, ignoring loaded); the payload-level inaccuracy is issue #158. Dropping the default is a behaviour change \u2014 machine-as-brain boxes rely on it to wire senses.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "fact", "record_metadata": {"topic": "compose-defaults", "relates_to": "issue-158, PR-159", "box": "spark", "date": "2026-07-25"}, "created": "2026-07-25T09:00:09.357963+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 2.0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "spark-compose-bringup-needs-all-four-f-files", "hash": "26be8f3cf414b379cfd2ab52c32982ea261ecbd7b22e2b1c2c75d4da4e1bd92c", "content": "Spark ~/.lobes bring-up MUST name ALL FOUR compose files, not two. The previously-recorded rule 'always pass -f docker-compose.yml -f docker-compose.shape.yml' is INCOMPLETE and actively caused a live defect. Passing ANY explicit -f suppresses docker compose's auto-load of docker-compose.override.yml. Correct invocation: docker compose -f docker-compose.yml -f docker-compose.audio.yml -f docker-compose.shape.yml -f docker-compose.override.yml. MEASURED CONSEQUENCE (2026-07-31): services had been brought up with inconsistent overlay sets (gateway from 2 files, vllm-embed-deep from 4), so the running gateway carried ZERO EMBED_DEEP_* env vars and Qwen/Qwen3-Embedding-4B was ABSENT from /v1/models -- the 4B embed gear ran Up 4 days healthy holding GPU budget while completely unroutable. Diagnose with: docker inspect --format '{{index .Config.Labels \"com.docker.compose.project.config_files\"}}' -- if two containers show different file lists, the stack is split-brain. Fixed by re-scaffolding docker-compose.yml from the packaged fleet template (which carries vllm-embed-deep + EMBED_DEEP_* natively) and trimming override.yml to ONLY the stt loopback :9002 publish that reachy-mini-cli needs. Also: the shape.yml !resets gateway depends_on, so `up -d gateway` alone does NOT fan out to dropped services -- rebuilding just the gateway is safe on this box.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"box": "spark", "topic": "compose-overlays", "repo": "lobes-cli", "date": "2026-07-31"}, "created": "2026-07-31T14:41:45.072853+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 0.5, "links": ["spark-proxy-golive-senses-orin-muse-thor-20260717"], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "worker-peer-proxy-inert-before-0548", "hash": "97f2a557b8c41ec2a0da164668e7a5ab5d7d1ff9cf847308373d0f915f25003e", "content": "lobes worker peer-proxy was SILENTLY INERT from 0.54.6 (when the worker role landed) until the 0.54.8 fix. lobes/gateway/_config.py's PEER_PROXY_ENV carries eight roles; lobes/gateway/server.py's _PEER_SERVED_NAME_ENV and _PEER_ROLE_HINT carried only the five pre-worker ones. _peer_served_name therefore returned '' for worker EVEN WITH WORKER_SERVED_NAME set (that var was never looked up), so peer_specs_from_table dropped the role at its 'if not served_name: continue' guard -> no ReadinessCache peer probe, no /v1/models entry, no _proxied_owner match, so WORKER_PEER_PROXY=true forwarded nothing and fell through to the referral-only role_infeasible 404. It hit exactly the shape that needs it: worker is an opt-in core role, so only a worker-hosting shape renders WORKER_BASE_URL; a box that merely REACHES worker has no wired Backend to resolve off and lands on the two broken tables. CI COULD NOT SEE IT: every pre-existing worker proxy test used _worker_dropped_env(), which sets WORKER_BASE_URL -- a wired-but-infeasible shape that resolves via the wired backend and never touches either table; that helper's own docstring concluded 'no server.py change required', true only for the shape it tested. LESSON: when adding a role to PEER_PROXY_ENV, also teach BOTH server.py tables, and test the UNWIRED shape. Guard now exists: tests/test_gateway_proxy.py::test_every_proxyable_role_resolves_a_served_name iterates PEER_PROXY_ENV. Found by a devague falsification task whose whole job was to DISPROVE 'the proxy needs no lobes/ change'.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"topic": "proxy-lobes", "repo": "lobes-cli", "relates_to": "PR-165, issue-115, issue-127", "date": "2026-07-31"}, "created": "2026-07-31T14:41:45.075714+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 1.0, "links": ["spark-compose-bringup-needs-all-four-f-files"], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "spark-proxy-advert-refresh-20260805", "hash": "90cb3699ea4df37e330bdc84a7ffbd7ac188a7efe3151ccdbddbb6d33391459b", "content": "Spark GB10 ~/.lobes proxy advertisement refreshed 2026-08-05 (lobes-cli 0.55.0): lobes capabilities now shows the TRUE mesh \u2014 senses by-proxy from orin as unsloth/gemma-4-12B-it-qat-w4a16 @262144 ready=true, worker by-proxy from thor as unsloth/Qwen3.6-35B-A3B-NVFP4 ready=true, muse dormant (infeasible, hosted_by=null). TWO DISTINCT ROOT CAUSES for the prior wrong display. (1) STALE SERVED NAME: the orin re-served senses on the unsloth QAT W4A16 checkpoint (0.55.0 Orin-card promotion) but the Spark .env still had MULTIMODAL_SERVED_NAME=coolthor/gemma-4-12B-it-NVFP4A16 \u2014 the peer-readiness probe looks for the ADVERTISED id in the peer /v1/models, so senses sat proxied=true/ready=false and /v1/models advertised a model the orin no longer serves. A peer checkpoint swap MUST be mirrored into every proxying box .env served-name. (2) CLI-RENDER GAP: the missing worker row was purely the INSTALLED CLI (uv tool, 0.54.5, predates the 0.54.6 worker role) \u2014 the 0.54.8 gateway payload already had worker proxied+ready; `lobes capabilities` renders from the gateway payload but the old CLI dropped the unknown role. Check `lobes --version` vs the gateway pin before diagnosing the gateway. muse needed NO change (referral already removed from .env 2026-07-31; payload honest; a real model=muse request 404s role_infeasible hosted_by=null). CHANGES: .env MULTIMODAL_MODEL+SERVED_NAME -> unsloth/gemma-4-12B-it-qat-w4a16; MULTIMODAL_MAX_MODEL_LEN 32768 -> 262144 (mirrors what the orin peer serves \u2014 inert locally since the shape drops vllm-multimodal; re-derive before any local re-host, 262144@0.14 never booted on GB10); MODEL_GEAR_VERSION 0.54.8 -> 0.55.0 (gateway image rebuilt, verified 0.55.0 inside); VLLM_MODEL/VLLM_SERVED_NAME coherence mirrors -> unsloth/Qwen3.6-27B-NVFP4 (file rule: keep equal to PRIMARY_*); uv tool upgrade lobes-cli 0.54.5 -> 0.55.0. Gateway recreated with the four -f files + --no-deps (no other container touched). VERIFIED LIVE: senses 200 X-Lobes-Proxied-By orin reply exact; worker 200 X-Lobes-Proxied-By thor (NOTE: worker thinking trace comes back in a `reasoning` field and a small max_tokens is consumed by it \u2014 first probe at 200 tokens returned content=null finish=length; 800 tokens returned the exact reply); muse 404 role_infeasible.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "fact", "record_metadata": {"box": "spark", "topic": "proxy-advertisement-refresh", "relates_to": "PR-176, issue-115, issue-127", "date": "2026-08-05"}, "created": "2026-08-05T19:08:29.450418+00:00", "last_recall": null, "recall_count": 0, "links": ["spark-proxy-golive-senses-orin-muse-thor-20260717", "worker-peer-proxy-inert-before-0548", "spark-compose-bringup-needs-all-four-f-files"], "supersedes": null, "lifecycle": "active", "added_by": null}} diff --git a/AGENTS.md b/AGENTS.md index a0d21ff2..0474f8b3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,11 +36,14 @@ The division of labor: daria notices, **you reason**, lecodeur builds. ## Runtime You are served by a locally-hosted vLLM reasoning model -(`sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP` — a Qwen3.6 27B with hybrid -Mamba/linear-attention layers and a grafted MTP draft head for speculative -decoding, text-only, in NVFP4, 256K native context served at the full 256K on the -shared DGX Spark; ~2.4x decode over the archived baseline) over the +(`unsloth/Qwen3.6-27B-NVFP4` — a Qwen3.6 27B with hybrid +Mamba/linear-attention layers and a self-hosted MTP draft head baked into the +checkpoint for speculative decoding, in compressed-tensors NVFP4, 256K native +context served at the full 256K on the shared DGX Spark) over the `acp` backend — not a Claude-backed runtime. It has a thinking mode and emits a reasoning trace before -its answer, which suits a deep thinker. This file is your system prompt; +its answer, which suits a deep thinker. It is also **multimodal** — you take +image and video input through the checkpoint's own ViT, so "look at this +screenshot" is something you can actually do, not something to hand off. This +file is your system prompt; `CLAUDE.md` is separate guidance for a Claude that resides in the repo to help build and maintain it. diff --git a/CHANGELOG.md b/CHANGELOG.md index 7fa4aceb..374d390d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,21 @@ All notable changes to this project are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.55.1] - 2026-08-10 + +**Three surfaces still named the checkpoint 0.54.9 demoted.** The multimodal +`cortex` promotion swapped the served id to `unsloth/Qwen3.6-27B-NVFP4` and left +`sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP` behind as a text-only *candidate* — +but `whoami`'s fallback, the agent's own system prompt, and two README passages +kept describing the demoted one as what this box serves. Docs-and-fallback only; +no runtime routing changed. + +### Fixed + +- **`lobes whoami` reported a model the box 404s on.** `_DEFAULT_MODEL` — what `whoami` prints as "served" when there is no scaffold, or when `VLLM_SERVED_NAME` is unset — still named the demoted `sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP`, while the gateway's own fallback (`lobes.gateway._config._DEFAULT_PRIMARY`) had moved to `unsloth/Qwen3.6-27B-NVFP4` in 0.54.9. On an unscaffolded box the two answers disagreed, and `whoami`'s was the wrong one: a caller pinning the id it printed gets `model_not_found`. The two constants must track each other; the comment now says so. +- **`AGENTS.md` told the deployed agent it was text-only.** The runtime paragraph — the `acp` system prompt the lobes agent actually runs on — described the demoted checkpoint, its grafted MTP head, and a text-only capability set. It now describes the promoted one (self-hosted MTP baked into the checkpoint, compressed-tensors NVFP4) and states the capability that changed: the agent takes **image and video** input through the checkpoint's own ViT, so "look at this screenshot" is work it can do rather than hand off. An agent that does not know it can see will keep referring vision away. +- **`README.md` taught callers the addressing pattern that just broke.** The fleet-routing example pinned a raw checkpoint id — the exact id 0.54.9 retired — so copying it yielded a 404; it now addresses the stable role name (`model: "cortex"`) and says why. The example's comment also claimed an unknown model "defaults to the primary"; only a **missing** `model` field does that, an unknown id 404s `model_not_found`. The `culture.yaml` snippet in the identity section was updated to the served id as well. + ## [0.55.0] - 2026-08-04 **The Jetson AGX Orin becomes a first-class card.** `orin` joins `spark`/`thor` diff --git a/README.md b/README.md index 86fa6425..82cdca3c 100644 --- a/README.md +++ b/README.md @@ -238,8 +238,10 @@ lobes fleet status # container states + gateway /health + /v1/mod ```bash curl -s http://localhost:8000/v1/models # the WARM backend(s) (not the full catalog — see below) -# an unknown/missing model defaults to the primary; route explicitly by name: -curl -s http://localhost:8000/v1/chat/completions -d '{"model":"sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP","messages":[...]}' +# a MISSING model field defaults to the primary; an UNKNOWN id 404s (model_not_found). +# Address the fleet by stable role/tier name — never a raw checkpoint id, which a +# model swap retires out from under you (0.54.9 did exactly that): +curl -s http://localhost:8000/v1/chat/completions -d '{"model":"cortex","messages":[...]}' ``` The fleet runs a **default-on `cortex` + `senses` duo** (the `main` + `multimodal` @@ -425,7 +427,7 @@ auth/exposure — or `lobes explain api`. `lobes` is one identity, not two: it is the repo/tool that serves the model *and* the local thinking agent deployed on it. The agent's runtime identity lives in `AGENTS.md` (the `acp` system prompt) and `culture.yaml` (`suffix: lobes`, -`backend: acp`, `model: vllm-local/sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP`) — the same +`backend: acp`, `model: vllm-local/unsloth/Qwen3.6-27B-NVFP4`) — the same lobes that runs the engine consumes it over the `acp` `vllm-local` provider. ## Acknowledgements diff --git a/lobes/cli/_commands/whoami.py b/lobes/cli/_commands/whoami.py index 55806bb5..38cf82af 100644 --- a/lobes/cli/_commands/whoami.py +++ b/lobes/cli/_commands/whoami.py @@ -18,7 +18,11 @@ from lobes.runtime import _compose, _env, _health _FALLBACK_AGENT = "lobes" -_DEFAULT_MODEL = "sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP" +# What `whoami` reports as "served" when there is no scaffold, or when +# VLLM_SERVED_NAME is unset. Must track the gateway's own fallback +# (``lobes.gateway._config._DEFAULT_PRIMARY``) — reporting a DEMOTED checkpoint +# here tells an operator the box serves a model it 404s on. +_DEFAULT_MODEL = "unsloth/Qwen3.6-27B-NVFP4" def _find_culture_yaml() -> Path | None: diff --git a/pyproject.toml b/pyproject.toml index 9f78eead..b02f26cc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "lobes-cli" -version = "0.55.0" +version = "0.55.1" description = "lobes — run, assess, and switch the local vLLM model." readme = "README.md" license = "Apache-2.0" diff --git a/uv.lock b/uv.lock index ce8dd4e0..30508043 100644 --- a/uv.lock +++ b/uv.lock @@ -440,7 +440,7 @@ wheels = [ [[package]] name = "lobes-cli" -version = "0.54.9" +version = "0.55.1" source = { editable = "." } [package.optional-dependencies]