Skip to content

fix: three surfaces still named the checkpoint 0.54.9 demoted (0.55.1) - #179

Merged
OriNachum merged 1 commit into
mainfrom
fix/stale-demoted-checkpoint-references
Aug 9, 2026
Merged

fix: three surfaces still named the checkpoint 0.54.9 demoted (0.55.1)#179
OriNachum merged 1 commit into
mainfrom
fix/stale-demoted-checkpoint-references

Conversation

@OriNachum

Copy link
Copy Markdown
Contributor

What

The 0.54.9 multimodal cortex promotion swapped the served id to
unsloth/Qwen3.6-27B-NVFP4 and demoted
sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP to a text-only candidate
(cite-don't-delete). Three surfaces kept describing the demoted checkpoint as
what this box serves.

Docs and fallback constants only — no runtime routing changed.

The three

surface was why it matters
lobes/cli/_commands/whoami.py_DEFAULT_MODEL the demoted id It is what whoami prints as "served" with no scaffold, or VLLM_SERVED_NAME unset. The gateway's own fallback (lobes.gateway._config._DEFAULT_PRIMARY) moved in 0.54.9, so on an unscaffolded box the two disagreed — and whoami's answer was the wrong one. A caller pinning what it printed gets model_not_found. The two constants must track each other; a comment now says so.
AGENTS.md — the Runtime paragraph a text-only runtime with a grafted MTP head This file is the deployed lobes agent's actual acp system prompt. It now describes the promoted checkpoint (self-hosted MTP baked in, compressed-tensors NVFP4) and states the capability that changed: image and video intake through the checkpoint's own ViT. An agent that does not know it can see keeps referring vision away.
README.md — the fleet-routing example + culture.yaml snippet a raw pinned checkpoint id The example pinned the exact id 0.54.9 retired, so copying it yields a 404. It now addresses the stable role name (model: "cortex") and says why. The neighbouring comment also claimed an unknown model "defaults to the primary" — only a missing model field does that; an unknown id 404s model_not_found (gateway/server.py:1354).

Also lands eidetic recall bookkeeping (recall-count bumps on 13 records) and one
new public record, spark-proxy-advert-refresh-20260805.

Scope, and what is deliberately left

Other stale references to the demoted id remain in docs/gateway-fleet.md,
openai-api.md, colleague-stack.md, machine-profiles.md, and
lobes/explain/catalog.py still show it as the default primary in tables and
examples. That is a wider docs sweep, out of scope for this fix; the three here
are the ones that either mislead an operator at runtime (whoami), misinform
the agent about its own capabilities (AGENTS.md), or hand a caller a request
that 404s (README.md). Test fixtures and tests/* constants keep the old id
on purpose — they pin behaviour, not the current default.

Verification

  • uv run pytest -n auto -q — 2867 passed, 15 skipped

  • black / isort / flake8 / bandit — clean

  • markdownlint-cli2 CHANGELOG.md README.md AGENTS.md — 0 errors

  • uv run afi cli doctor . --strict — pass

  • Version bumped 0.55.0 → 0.55.1 (CHANGELOG entry filled in — it was committed as an empty skeleton)

  • lobes (Claude)

🤖 Generated with Claude Code

https://claude.ai/code/session_01Eq2ZEyb6PV9ntkSzfsuRsm

The multimodal `cortex` promotion swapped the served id to
`unsloth/Qwen3.6-27B-NVFP4` and left
`sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP` behind as a text-only candidate.
Three surfaces kept describing the demoted one as what this box serves:

- `lobes whoami`'s `_DEFAULT_MODEL` fallback — the id it prints when there
  is no scaffold or `VLLM_SERVED_NAME` is unset — disagreed with the
  gateway's own `_DEFAULT_PRIMARY`, and `whoami`'s answer was the wrong
  one: a caller pinning what it printed gets `model_not_found`.
- `AGENTS.md`, the deployed agent's actual `acp` system prompt, described
  a text-only runtime with a grafted MTP head. An agent that does not know
  it can see keeps referring vision away, so the paragraph now states the
  image/video intake through the checkpoint's own ViT.
- `README.md` taught the addressing pattern that just broke — an example
  pinning the retired raw id. It now addresses the stable `cortex` role
  name, and the neighbouring comment no longer claims an unknown model
  falls back to the primary (only a *missing* `model` field does; an
  unknown id 404s).

Also lands eidetic recall bookkeeping and one new public record,
`spark-proxy-advert-refresh-20260805`.

Docs and fallback constants only — no runtime routing changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eq2ZEyb6PV9ntkSzfsuRsm
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Fix stale cortex checkpoint references after 0.54.9 promotion (0.55.1)

🐞 Bug fix 📝 Documentation ⚙️ Configuration changes 🕐 20-40 Minutes

Grey Divider

AI Description

• Align lobes whoami fallback model with the gateway’s primary default to avoid 404s.
• Update agent prompt and README examples to reference the promoted cortex checkpoint/role.
• Bump version to 0.55.1 and record the fix in CHANGELOG + eidetic public memory.
Diagram

graph TD
  U(["Operator / Caller"]) --> W["whoami.py default model"] --> M["Served model id: unsloth/Qwen3.6-27B-NVFP4"]
  U --> A["AGENTS.md runtime prompt"] --> M
  U --> R["README.md routing examples"] --> M
  W -. "tracks" .-> G["gateway _DEFAULT_PRIMARY"] --> M
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Single source of truth for default primary model
  • ➕ Eliminates drift between gateway defaults, CLI fallbacks, and docs
  • ➕ Makes future promotions/demotions a one-line change
  • ➖ May introduce an undesirable dependency edge (CLI importing gateway internals) unless a shared constants module is introduced
2. Derive “served model” dynamically from `/v1/models`
  • ➕ Always reports what’s actually reachable at runtime
  • ➕ Avoids hard-coded fallbacks going stale
  • ➖ Requires gateway reachability; whoami may be used before networking/containers are up
  • ➖ Adds latency and error-handling complexity for a simple status command

Recommendation: The PR’s current approach is appropriate for a patch release (fix the few high-impact stale surfaces). As a follow-up, consider introducing a small shared constant/module for the default primary model to prevent future drift without coupling CLI to gateway internals.

Files changed (7) +49 / -24

Bug fix (1) +5 / -1
whoami.pyAlign 'whoami' fallback model with gateway default primary +5/-1

Align 'whoami' fallback model with gateway default primary

• Changes '_DEFAULT_MODEL' from the demoted checkpoint to 'unsloth/Qwen3.6-27B-NVFP4'. Adds a comment documenting that this fallback must track 'lobes.gateway._config._DEFAULT_PRIMARY' to avoid misleading operators into pinning a non-existent model id.

lobes/cli/_commands/whoami.py

Documentation (4) +42 / -21
lobes__public.jsonlRefresh eidetic recall bookkeeping and add a new public record +14/-13

Refresh eidetic recall bookkeeping and add a new public record

• Bumps recall metadata on multiple existing records and adds the new 'spark-proxy-advert-refresh-20260805' entry. No runtime code behavior changes; this is repo bookkeeping/reference material.

.eidetic/memory/lobes__public.jsonl

AGENTS.mdUpdate deployed agent runtime identity to promoted cortex checkpoint +8/-5

Update deployed agent runtime identity to promoted cortex checkpoint

• Replaces the demoted checkpoint reference with 'unsloth/Qwen3.6-27B-NVFP4' and updates the description to reflect baked-in MTP + compressed-tensors NVFP4. Clarifies that the agent is multimodal (image/video via the checkpoint ViT).

AGENTS.md

CHANGELOG.mdAdd 0.55.1 release notes for stale checkpoint reference fixes +15/-0

Add 0.55.1 release notes for stale checkpoint reference fixes

• Adds a 0.55.1 entry documenting the three corrected surfaces (whoami fallback, AGENTS prompt, README examples) and clarifying that routing behavior is unchanged.

CHANGELOG.md

README.mdFix routing examples to use stable role name and correct 404 semantics +5/-3

Fix routing examples to use stable role name and correct 404 semantics

• Updates the fleet routing example to use 'model: "cortex"' instead of a retired checkpoint id and clarifies that missing 'model' defaults, but unknown ids 404 'model_not_found'. Updates the identity snippet to the promoted 'vllm-local/unsloth/Qwen3.6-27B-NVFP4' model value.

README.md

Other (2) +2 / -2
pyproject.tomlBump package version to 0.55.1 +1/-1

Bump package version to 0.55.1

• Updates the project version from 0.55.0 to 0.55.1 to ship the documentation/fallback corrections.

pyproject.toml

uv.lockUpdate lockfile for local package version bump +1/-1

Update lockfile for local package version bump

• Reflects the 'lobes-cli' editable package version moving to 0.55.1 in the uv lockfile.

uv.lock

@sonarqubecloud

sonarqubecloud Bot commented Aug 9, 2026

Copy link
Copy Markdown

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. Fractional recall_count values 🐞 Bug ≡ Correctness
Description
.eidetic/memory/lobes__public.jsonl now contains non-integer recall_count values (e.g. 5.5),
which contradicts the documented meaning of recall_count as a count of recalls and can
break/confuse any consumer that assumes an integer count.
Code

.eidetic/memory/lobes__public.jsonl[35]

+{"id": "unified-memory-budget-is-measured-truth-2026-07-14", "hash": "b6258062566bba19a3de9fad95bb4eba773be9b6b233ae69267473f566b2cf6f", "content": "Brain-shapes live validation (#113, 2026-07-14) PROVED shape budgets on unified-memory boxes are MEASURED truths, not arithmetic. vLLM checks free-at-boot >= util*total: GB10 spark-lobe cortex REFUSED util 0.60 (needs 73.01 GiB; only 59.35/121.69 GiB free \u2014 host OS, eidetic/nova stacks, page cache share the pool) -> shipped 0.44 (reclaim-sum 0.30+0.14) which boots healthy at FULL native 262144 context: KV pool 888,946 tokens, 3.39x concurrency (hybrid Mamba Qwen3.6 = cheap long-context KV). Thor thor-lobe senses REFUSED even the 0.44 reclaim-sum (needs 54.04; only 38.44/122.82 free \u2014 heavier host workloads) -> shipped 0.30 (dropped cortex share only) at full 131072: KV pool 1,418,554 tokens, 10.82x concurrency (Gemma sliding-window layers cap long-context KV). Both PASS transcripts in docs/evidence/. LESSON: the acceptance run (scripts/accept-shape.sh) is what validates a shape budget on a box; declared reclaim values are hypotheses until a physical boot confirms.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "shape-budgets-measured", "relates_to": "issue-113-112", "date": "2026-07-14"}, "created": "2026-07-14T09:47:24.654680+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 5.5, "links": ["issue-112-one-lobe-per-box-specced"], "supersedes": null, "lifecycle": "active", "added_by": null}}
Relevance

●● Moderate

Recall bookkeeping is routinely committed; fractional counts might be intentional weighting vs stale
docs—unclear.

PR-#164

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The recall skill docs define recall_count as the number of times a record has been recalled, but
the updated JSONL record shows recall_count with a fractional value (5.5), which is not a valid
“number of times”.

.claude/skills/recall/SKILL.md[74-90]
.eidetic/memory/lobes__public.jsonl[35-41]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
Several public eidetic memory records now have *fractional* `recall_count` values (e.g. `5.5`, `8.5`, `2.5`, `0.5`). This contradicts the documented meaning of `recall_count` as the **number of times** a record has been recalled.

### Issue Context
The repo’s recall skill documentation treats `recall_count` as a count (an integer concept). Fractional values introduce an implicit new meaning (“weighted recall”), but the schema/docs/consumers are not updated to reflect that.

### Fix Focus Areas
- .eidetic/memory/lobes__public.jsonl[35-41]
- .claude/skills/recall/SKILL.md[74-90]

### Proposed fix
- Update the affected records’ `recall_count` values to integers (e.g. `5.5 -> 6`, `2.5 -> 3`, etc.).
- If fractional reinforcement is intentional, introduce a separate field (e.g. `recall_weight`) and keep `recall_count` as an integer, and update the docs accordingly.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context used
✅ Compliance rules (platform): 95 rules

Grey Divider

Tip of the day
💡 Did you know, you can ask Qodo to dismiss a finding you disagree with, with your reason on record

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

{"id": "lobes-pressure-swap-occupancy-sticky-2026-07-09", "hash": "7f666a9ea25527288cadc52f8b87e5499164ea353ffed40d2e3eda9aa7868e6b", "content": "lobes pressure policy sheds 100% of generate traffic FOREVER on a box whose swap has ever filled (filed as issue #100). lobes/runtime/_pressure.py computes swap_used_percent = (SwapTotal - SwapFree)/SwapTotal from /proc/meminfo \u2014 an OCCUPANCY metric. Swap occupancy is STICKY: the kernel does not reclaim swapped pages just because RAM freed, so once it crosses the 75% default threshold it stays there until swapoff/swapon. On the DGX Spark GB10 (2026-07-09): swap 16322/16383 MB = 99.6% occupancy, BUT `vmstat` si=0 so=0 (nothing being swapped) and /proc/pressure/memory shows `full avg10=0.00 avg60=0.00` (ZERO memory stall), with 26 GB RAM available. Since #88 (0.37.0) the policy SHEDS with 429 server_busy instead of degrading, so every cortex/senses request 429s while `X-Lobes-Override: 1` still returns 200. The rig had been fine on gateway 0.36.0 (pre-shed) and broke the instant it was upgraded to 0.39.0 \u2014 the wire had been armed for weeks. WORKAROUND (documented, but it DISABLES the policy rather than fixing it): set LOBES_SWAP_DEGRADED_THRESHOLD=100 and LOBES_IOWAIT_DEGRADED_THRESHOLD=100 in the deployment .env and recreate the gateway. Note LOBES_IOWAIT_DEGRADED_THRESHOLD=100 was ALREADY set by a previous operator for the same class of reason (0.36.2 CHANGELOG documents \"phantom high iowait on an idle disk, e.g. the DGX Spark GB10\") \u2014 half the policy had already been switched off, which is the tell that the METRIC is wrong, not the threshold. RIGHT FIX: read PSI (/proc/pressure/memory, /proc/pressure/io avg10) or swap RATE (pswpin/pswpout deltas from /proc/vmstat), not occupancy; keep occupancy at most as a secondary AND-condition. Also: a gateway that has shed 100% of requests for an hour is misconfigured, not busy \u2014 nothing currently warns.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "pressure-policy", "relates_to": "issues-100-85-86-88", "date": "2026-07-09"}, "created": "2026-07-09T09:38:15.369909+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "lobes-fleet-frozen-image-pin-2026-07-09", "hash": "08f35f73d1ad1f81fc3dd0a6d348a2d8f7f6c2ef4581caa92dfba6c7a56b14c5", "content": "lobes-cli fleet deployments FREEZE their lobes version and nothing bumps it (issue #99). Dockerfile.gateway/realtime/stt/chatterbox all run: pip install --no-cache-dir lobes-cli==${MODEL_GEAR_VERSION}. `lobes init` writes MODEL_GEAR_VERSION into .env ONCE at scaffold time and no verb ever rewrites it, so `docker compose up -d --build gateway` faithfully reinstalls the OLD pinned version. On 2026-07-09 the reference rig ran THREE lobes versions at once: gateway 0.36.0 (image built 07-03), realtime 0.34.1 (07-02), stt/chatterbox 0.34.1-era (07-02) \u2014 against a CLI at 0.39.0, with 0.36.1/0.36.2/0.37.0/0.38.0/0.39.0 ALL published on PyPI. CONSEQUENCE: three bugs were reported against lobes CODE that were merely undeployed fixes. (a) #92 'advertised endpoint :8000 404s' \u2014 reachable_origin shipped in 0.38.0/PR#90 on 07-04; the gateway image predated it by one day, so it fell back to _gateway_base_url() = its INTERNAL container port 8000 while the published host port is 8001. Worse, host :8000 was an unrelated uvicorn service (reachy-mini-dae), so the advertised endpoint pointed at a foreign daemon. (b) the stt sidecar answered /v1/health/ready with 200 status=ready while its CUDA context was poisoned and transcription 500d \u2014 #89 (07-04) had replaced the old liveness-only handler with a model_loaded+cuda_ok probe, but the stt image was built 07-02. (c) chatterbox likewise. CHEAP FIX (no model reload): edit ~/.lobes/.env MODEL_GEAR_VERSION=<cli version>, then from ~/.lobes run: docker compose -f docker-compose.yml -f docker-compose.audio.yml up -d --build --no-deps gateway realtime stt chatterbox. GOTCHA 1: the compose PROJECT is named lobes (from the dir basename) while containers are named model-gear-* via container_name \u2014 check the com.docker.compose.project label before invoking compose or you create a duplicate project. GOTCHA 2: AUDIO_URL reaches the gateway ONLY via docker-compose.audio.yml, so a gateway recreated without the -f audio overlay silently loses the audio lane (issue #96). GOTCHA 3: a from-source checkout can be AHEAD of PyPI, in which case the gateway image cannot be built from the pin at all \u2014 the dev path needs a locally built wheel COPYd in.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "fleet-version-pin", "relates_to": "issues-99-92-89-96", "date": "2026-07-09"}, "created": "2026-07-09T09:38:41.844728+00:00", "last_recall": "2026-07-24T18:41:34.170386+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "unified-memory-budget-is-measured-truth-2026-07-14", "hash": "b6258062566bba19a3de9fad95bb4eba773be9b6b233ae69267473f566b2cf6f", "content": "Brain-shapes live validation (#113, 2026-07-14) PROVED shape budgets on unified-memory boxes are MEASURED truths, not arithmetic. vLLM checks free-at-boot >= util*total: GB10 spark-lobe cortex REFUSED util 0.60 (needs 73.01 GiB; only 59.35/121.69 GiB free \u2014 host OS, eidetic/nova stacks, page cache share the pool) -> shipped 0.44 (reclaim-sum 0.30+0.14) which boots healthy at FULL native 262144 context: KV pool 888,946 tokens, 3.39x concurrency (hybrid Mamba Qwen3.6 = cheap long-context KV). Thor thor-lobe senses REFUSED even the 0.44 reclaim-sum (needs 54.04; only 38.44/122.82 free \u2014 heavier host workloads) -> shipped 0.30 (dropped cortex share only) at full 131072: KV pool 1,418,554 tokens, 10.82x concurrency (Gemma sliding-window layers cap long-context KV). Both PASS transcripts in docs/evidence/. LESSON: the acceptance run (scripts/accept-shape.sh) is what validates a shape budget on a box; declared reclaim values are hypotheses until a physical boot confirms.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "shape-budgets-measured", "relates_to": "issue-113-112", "date": "2026-07-14"}, "created": "2026-07-14T09:47:24.654680+00:00", "last_recall": "2026-07-31T13:22:52.104978+00:00", "recall_count": 5.0, "links": ["issue-112-one-lobe-per-box-specced"], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "unified-memory-budget-is-measured-truth-2026-07-14", "hash": "b6258062566bba19a3de9fad95bb4eba773be9b6b233ae69267473f566b2cf6f", "content": "Brain-shapes live validation (#113, 2026-07-14) PROVED shape budgets on unified-memory boxes are MEASURED truths, not arithmetic. vLLM checks free-at-boot >= util*total: GB10 spark-lobe cortex REFUSED util 0.60 (needs 73.01 GiB; only 59.35/121.69 GiB free \u2014 host OS, eidetic/nova stacks, page cache share the pool) -> shipped 0.44 (reclaim-sum 0.30+0.14) which boots healthy at FULL native 262144 context: KV pool 888,946 tokens, 3.39x concurrency (hybrid Mamba Qwen3.6 = cheap long-context KV). Thor thor-lobe senses REFUSED even the 0.44 reclaim-sum (needs 54.04; only 38.44/122.82 free \u2014 heavier host workloads) -> shipped 0.30 (dropped cortex share only) at full 131072: KV pool 1,418,554 tokens, 10.82x concurrency (Gemma sliding-window layers cap long-context KV). Both PASS transcripts in docs/evidence/. LESSON: the acceptance run (scripts/accept-shape.sh) is what validates a shape budget on a box; declared reclaim values are hypotheses until a physical boot confirms.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "shape-budgets-measured", "relates_to": "issue-113-112", "date": "2026-07-14"}, "created": "2026-07-14T09:47:24.654680+00:00", "last_recall": "2026-08-05T19:00:48.654992+00:00", "recall_count": 5.5, "links": ["issue-112-one-lobe-per-box-specced"], "supersedes": null, "lifecycle": "active", "added_by": null}}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

1. Fractional recall_count values 🐞 Bug ≡ Correctness

.eidetic/memory/lobes__public.jsonl now contains non-integer recall_count values (e.g. 5.5),
which contradicts the documented meaning of recall_count as a count of recalls and can
break/confuse any consumer that assumes an integer count.
Agent Prompt
### Issue description
Several public eidetic memory records now have *fractional* `recall_count` values (e.g. `5.5`, `8.5`, `2.5`, `0.5`). This contradicts the documented meaning of `recall_count` as the **number of times** a record has been recalled.

### Issue Context
The repo’s recall skill documentation treats `recall_count` as a count (an integer concept). Fractional values introduce an implicit new meaning (“weighted recall”), but the schema/docs/consumers are not updated to reflect that.

### Fix Focus Areas
- .eidetic/memory/lobes__public.jsonl[35-41]
- .claude/skills/recall/SKILL.md[74-90]

### Proposed fix
- Update the affected records’ `recall_count` values to integers (e.g. `5.5 -> 6`, `2.5 -> 3`, etc.).
- If fractional reinforcement is intentional, introduce a separate field (e.g. `recall_weight`) and keep `recall_count` as an integer, and update the docs accordingly.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

@OriNachum
OriNachum merged commit bb8b8b7 into main Aug 9, 2026
10 checks passed
@OriNachum
OriNachum deleted the fix/stale-demoted-checkpoint-references branch August 9, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant