fix: three surfaces still named the checkpoint 0.54.9 demoted (0.55.1) - #179
Conversation
The multimodal `cortex` promotion swapped the served id to `unsloth/Qwen3.6-27B-NVFP4` and left `sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP` behind as a text-only candidate. Three surfaces kept describing the demoted one as what this box serves: - `lobes whoami`'s `_DEFAULT_MODEL` fallback — the id it prints when there is no scaffold or `VLLM_SERVED_NAME` is unset — disagreed with the gateway's own `_DEFAULT_PRIMARY`, and `whoami`'s answer was the wrong one: a caller pinning what it printed gets `model_not_found`. - `AGENTS.md`, the deployed agent's actual `acp` system prompt, described a text-only runtime with a grafted MTP head. An agent that does not know it can see keeps referring vision away, so the paragraph now states the image/video intake through the checkpoint's own ViT. - `README.md` taught the addressing pattern that just broke — an example pinning the retired raw id. It now addresses the stable `cortex` role name, and the neighbouring comment no longer claims an unknown model falls back to the primary (only a *missing* `model` field does; an unknown id 404s). Also lands eidetic recall bookkeeping and one new public record, `spark-proxy-advert-refresh-20260805`. Docs and fallback constants only — no runtime routing changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eq2ZEyb6PV9ntkSzfsuRsm
PR Summary by QodoFix stale cortex checkpoint references after 0.54.9 promotion (0.55.1)
AI Description
Diagram
High-Level Assessment
Files changed (7)
|
|
Code Review by Qodo
1. Fractional recall_count values
|
| {"id": "lobes-pressure-swap-occupancy-sticky-2026-07-09", "hash": "7f666a9ea25527288cadc52f8b87e5499164ea353ffed40d2e3eda9aa7868e6b", "content": "lobes pressure policy sheds 100% of generate traffic FOREVER on a box whose swap has ever filled (filed as issue #100). lobes/runtime/_pressure.py computes swap_used_percent = (SwapTotal - SwapFree)/SwapTotal from /proc/meminfo \u2014 an OCCUPANCY metric. Swap occupancy is STICKY: the kernel does not reclaim swapped pages just because RAM freed, so once it crosses the 75% default threshold it stays there until swapoff/swapon. On the DGX Spark GB10 (2026-07-09): swap 16322/16383 MB = 99.6% occupancy, BUT `vmstat` si=0 so=0 (nothing being swapped) and /proc/pressure/memory shows `full avg10=0.00 avg60=0.00` (ZERO memory stall), with 26 GB RAM available. Since #88 (0.37.0) the policy SHEDS with 429 server_busy instead of degrading, so every cortex/senses request 429s while `X-Lobes-Override: 1` still returns 200. The rig had been fine on gateway 0.36.0 (pre-shed) and broke the instant it was upgraded to 0.39.0 \u2014 the wire had been armed for weeks. WORKAROUND (documented, but it DISABLES the policy rather than fixing it): set LOBES_SWAP_DEGRADED_THRESHOLD=100 and LOBES_IOWAIT_DEGRADED_THRESHOLD=100 in the deployment .env and recreate the gateway. Note LOBES_IOWAIT_DEGRADED_THRESHOLD=100 was ALREADY set by a previous operator for the same class of reason (0.36.2 CHANGELOG documents \"phantom high iowait on an idle disk, e.g. the DGX Spark GB10\") \u2014 half the policy had already been switched off, which is the tell that the METRIC is wrong, not the threshold. RIGHT FIX: read PSI (/proc/pressure/memory, /proc/pressure/io avg10) or swap RATE (pswpin/pswpout deltas from /proc/vmstat), not occupancy; keep occupancy at most as a secondary AND-condition. Also: a gateway that has shed 100% of requests for an hour is misconfigured, not busy \u2014 nothing currently warns.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "pressure-policy", "relates_to": "issues-100-85-86-88", "date": "2026-07-09"}, "created": "2026-07-09T09:38:15.369909+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} | ||
| {"id": "lobes-fleet-frozen-image-pin-2026-07-09", "hash": "08f35f73d1ad1f81fc3dd0a6d348a2d8f7f6c2ef4581caa92dfba6c7a56b14c5", "content": "lobes-cli fleet deployments FREEZE their lobes version and nothing bumps it (issue #99). Dockerfile.gateway/realtime/stt/chatterbox all run: pip install --no-cache-dir lobes-cli==${MODEL_GEAR_VERSION}. `lobes init` writes MODEL_GEAR_VERSION into .env ONCE at scaffold time and no verb ever rewrites it, so `docker compose up -d --build gateway` faithfully reinstalls the OLD pinned version. On 2026-07-09 the reference rig ran THREE lobes versions at once: gateway 0.36.0 (image built 07-03), realtime 0.34.1 (07-02), stt/chatterbox 0.34.1-era (07-02) \u2014 against a CLI at 0.39.0, with 0.36.1/0.36.2/0.37.0/0.38.0/0.39.0 ALL published on PyPI. CONSEQUENCE: three bugs were reported against lobes CODE that were merely undeployed fixes. (a) #92 'advertised endpoint :8000 404s' \u2014 reachable_origin shipped in 0.38.0/PR#90 on 07-04; the gateway image predated it by one day, so it fell back to _gateway_base_url() = its INTERNAL container port 8000 while the published host port is 8001. Worse, host :8000 was an unrelated uvicorn service (reachy-mini-dae), so the advertised endpoint pointed at a foreign daemon. (b) the stt sidecar answered /v1/health/ready with 200 status=ready while its CUDA context was poisoned and transcription 500d \u2014 #89 (07-04) had replaced the old liveness-only handler with a model_loaded+cuda_ok probe, but the stt image was built 07-02. (c) chatterbox likewise. CHEAP FIX (no model reload): edit ~/.lobes/.env MODEL_GEAR_VERSION=<cli version>, then from ~/.lobes run: docker compose -f docker-compose.yml -f docker-compose.audio.yml up -d --build --no-deps gateway realtime stt chatterbox. GOTCHA 1: the compose PROJECT is named lobes (from the dir basename) while containers are named model-gear-* via container_name \u2014 check the com.docker.compose.project label before invoking compose or you create a duplicate project. GOTCHA 2: AUDIO_URL reaches the gateway ONLY via docker-compose.audio.yml, so a gateway recreated without the -f audio overlay silently loses the audio lane (issue #96). GOTCHA 3: a from-source checkout can be AHEAD of PyPI, in which case the gateway image cannot be built from the pin at all \u2014 the dev path needs a locally built wheel COPYd in.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "fleet-version-pin", "relates_to": "issues-99-92-89-96", "date": "2026-07-09"}, "created": "2026-07-09T09:38:41.844728+00:00", "last_recall": "2026-07-24T18:41:34.170386+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} | ||
| {"id": "unified-memory-budget-is-measured-truth-2026-07-14", "hash": "b6258062566bba19a3de9fad95bb4eba773be9b6b233ae69267473f566b2cf6f", "content": "Brain-shapes live validation (#113, 2026-07-14) PROVED shape budgets on unified-memory boxes are MEASURED truths, not arithmetic. vLLM checks free-at-boot >= util*total: GB10 spark-lobe cortex REFUSED util 0.60 (needs 73.01 GiB; only 59.35/121.69 GiB free \u2014 host OS, eidetic/nova stacks, page cache share the pool) -> shipped 0.44 (reclaim-sum 0.30+0.14) which boots healthy at FULL native 262144 context: KV pool 888,946 tokens, 3.39x concurrency (hybrid Mamba Qwen3.6 = cheap long-context KV). Thor thor-lobe senses REFUSED even the 0.44 reclaim-sum (needs 54.04; only 38.44/122.82 free \u2014 heavier host workloads) -> shipped 0.30 (dropped cortex share only) at full 131072: KV pool 1,418,554 tokens, 10.82x concurrency (Gemma sliding-window layers cap long-context KV). Both PASS transcripts in docs/evidence/. LESSON: the acceptance run (scripts/accept-shape.sh) is what validates a shape budget on a box; declared reclaim values are hypotheses until a physical boot confirms.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "shape-budgets-measured", "relates_to": "issue-113-112", "date": "2026-07-14"}, "created": "2026-07-14T09:47:24.654680+00:00", "last_recall": "2026-07-31T13:22:52.104978+00:00", "recall_count": 5.0, "links": ["issue-112-one-lobe-per-box-specced"], "supersedes": null, "lifecycle": "active", "added_by": null}} | ||
| {"id": "unified-memory-budget-is-measured-truth-2026-07-14", "hash": "b6258062566bba19a3de9fad95bb4eba773be9b6b233ae69267473f566b2cf6f", "content": "Brain-shapes live validation (#113, 2026-07-14) PROVED shape budgets on unified-memory boxes are MEASURED truths, not arithmetic. vLLM checks free-at-boot >= util*total: GB10 spark-lobe cortex REFUSED util 0.60 (needs 73.01 GiB; only 59.35/121.69 GiB free \u2014 host OS, eidetic/nova stacks, page cache share the pool) -> shipped 0.44 (reclaim-sum 0.30+0.14) which boots healthy at FULL native 262144 context: KV pool 888,946 tokens, 3.39x concurrency (hybrid Mamba Qwen3.6 = cheap long-context KV). Thor thor-lobe senses REFUSED even the 0.44 reclaim-sum (needs 54.04; only 38.44/122.82 free \u2014 heavier host workloads) -> shipped 0.30 (dropped cortex share only) at full 131072: KV pool 1,418,554 tokens, 10.82x concurrency (Gemma sliding-window layers cap long-context KV). Both PASS transcripts in docs/evidence/. LESSON: the acceptance run (scripts/accept-shape.sh) is what validates a shape budget on a box; declared reclaim values are hypotheses until a physical boot confirms.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "shape-budgets-measured", "relates_to": "issue-113-112", "date": "2026-07-14"}, "created": "2026-07-14T09:47:24.654680+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 5.5, "links": ["issue-112-one-lobe-per-box-specced"], "supersedes": null, "lifecycle": "active", "added_by": null}} |
There was a problem hiding this comment.
1. Fractional recall_count values 🐞 Bug ≡ Correctness
.eidetic/memory/lobes__public.jsonl now contains non-integer recall_count values (e.g. 5.5), which contradicts the documented meaning of recall_count as a count of recalls and can break/confuse any consumer that assumes an integer count.
Agent Prompt
### Issue description
Several public eidetic memory records now have *fractional* `recall_count` values (e.g. `5.5`, `8.5`, `2.5`, `0.5`). This contradicts the documented meaning of `recall_count` as the **number of times** a record has been recalled.
### Issue Context
The repo’s recall skill documentation treats `recall_count` as a count (an integer concept). Fractional values introduce an implicit new meaning (“weighted recall”), but the schema/docs/consumers are not updated to reflect that.
### Fix Focus Areas
- .eidetic/memory/lobes__public.jsonl[35-41]
- .claude/skills/recall/SKILL.md[74-90]
### Proposed fix
- Update the affected records’ `recall_count` values to integers (e.g. `5.5 -> 6`, `2.5 -> 3`, etc.).
- If fractional reinforcement is intentional, introduce a separate field (e.g. `recall_weight`) and keep `recall_count` as an integer, and update the docs accordingly.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



What
The 0.54.9 multimodal
cortexpromotion swapped the served id tounsloth/Qwen3.6-27B-NVFP4and demotedsakamakismile/Qwen3.6-27B-Text-NVFP4-MTPto a text-only candidate(cite-don't-delete). Three surfaces kept describing the demoted checkpoint as
what this box serves.
Docs and fallback constants only — no runtime routing changed.
The three
lobes/cli/_commands/whoami.py—_DEFAULT_MODELwhoamiprints as "served" with no scaffold, orVLLM_SERVED_NAMEunset. The gateway's own fallback (lobes.gateway._config._DEFAULT_PRIMARY) moved in 0.54.9, so on an unscaffolded box the two disagreed — andwhoami's answer was the wrong one. A caller pinning what it printed getsmodel_not_found. The two constants must track each other; a comment now says so.AGENTS.md— the Runtime paragraphacpsystem prompt. It now describes the promoted checkpoint (self-hosted MTP baked in, compressed-tensors NVFP4) and states the capability that changed: image and video intake through the checkpoint's own ViT. An agent that does not know it can see keeps referring vision away.README.md— the fleet-routing example +culture.yamlsnippetmodel: "cortex") and says why. The neighbouring comment also claimed an unknown model "defaults to the primary" — only a missingmodelfield does that; an unknown id 404smodel_not_found(gateway/server.py:1354).Also lands eidetic recall bookkeeping (recall-count bumps on 13 records) and one
new public record,
spark-proxy-advert-refresh-20260805.Scope, and what is deliberately left
Other stale references to the demoted id remain in
docs/—gateway-fleet.md,openai-api.md,colleague-stack.md,machine-profiles.md, andlobes/explain/catalog.pystill show it as the default primary in tables andexamples. That is a wider docs sweep, out of scope for this fix; the three here
are the ones that either mislead an operator at runtime (
whoami), misinformthe agent about its own capabilities (
AGENTS.md), or hand a caller a requestthat 404s (
README.md). Test fixtures andtests/*constants keep the old idon purpose — they pin behaviour, not the current default.
Verification
uv run pytest -n auto -q— 2867 passed, 15 skippedblack/isort/flake8/bandit— cleanmarkdownlint-cli2 CHANGELOG.md README.md AGENTS.md— 0 errorsuv run afi cli doctor . --strict— passVersion bumped 0.55.0 → 0.55.1 (CHANGELOG entry filled in — it was committed as an empty skeleton)
lobes (Claude)
🤖 Generated with Claude Code
https://claude.ai/code/session_01Eq2ZEyb6PV9ntkSzfsuRsm