Skip to content

perf(relay): keep the models.dev bodies off the relay - #3144

Merged
bobleer merged 1 commit into
GCWing:mainfrom
bobleer:lwb/relay-model-catalog-bodies
Sep 20, 2026
Merged

bobleer merged 1 commit into
GCWing:mainfrom
bobleer:lwb/relay-model-catalog-bodies

Conversation

@bobleer

@bobleer bobleer commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator

What

Every cross-machine model catalog read carried the full models.dev projections. The catalog that crosses a machine boundary now carries only the configured models, the defaults and the session selection. A controller that renders Model Settings while a peer is selected composes that surface from the host facts plus its own models.dev snapshot.

Why

models.dev is public data and every host refreshes its own snapshot, so the projections are identical by construction on both ends of a connection. Shipping them meant:

  • Model Settings opens, and a peer session poll, each moved a multi-MiB body the receiver already owned.
  • Session creation awaited the whole catalog before it even issued the session RPC (locally measured at ~4.4 s for get_ai_model_catalog, the dominant part of the "new session takes a long time" report).

Changes

  • RemoteModelCatalog is now built without the models.dev bodies on every cross-machine path: the peer host answer, the remote session poll, the mobile and bot controllers, and the CLI peer host.
  • The slim build still reports the built-in provider catalog revision, so RemoteModelCatalog::version does not move. A controller compares that version, so a slim build that dropped the revision would have made every poll look like a catalog change — provider_catalog.rs now exposes builtin_provider_catalog_identity(), and a test pins slim/full version equality plus revision sensitivity.
  • New controller-local get_local_models_dev_catalogs command serves the controller's own snapshot; get_ai_model_catalog keeps returning the full catalog only for in-process readers (TUI/app-server projections, plugin host) through get_local_model_catalog / get_remote_model_catalog.
  • Session creation reads three small config keys and asks for a single model's reasoning projection (project_ai_model_reasoning_catalog) instead of pulling the whole catalog.

Remote scenarios

Exercised in code and by tests for remote control (session poll path, bot router) and peer device mode (registry row, controller-local settings read). The remote-workspace path only consumes the same host trait, so it inherits the slim build.

Verification

  • cargo test -p openbitfun-services-integrations --no-default-features --features remote-connect (including the new slim_model_catalog_keeps_the_version_of_the_full_catalog, which fails if the slim build drops the revision)
  • core targeted tests (provider_catalog, reasoning catalog, bot routing)
  • cargo test -p openbitfun-product-domains
  • CLI peer host command tests; desktop command closure test (every command has one registry row)
  • web-ui flow_chat + infrastructure/config + infrastructure/api suites, type-check:web, type-check:mobile-web, eslint on the changed files
  • mobile test:device-compatibility, test:account-login
  • pnpm run check:core-boundaries, pnpm run capabilities:check, pnpm run fmt:rs

Notes

  • 1.0.2 does not need cross-version compatibility here: 1.0.2 clients and older clients do not share a relay, so the compatibility switches were removed rather than kept.
  • Known pre-existing gap, unchanged by this PR: the app-server (web) surface does not map get_ai_model_catalog, so a browser-hosted Model Settings page does not receive provider templates.

Co-authored-by: bitfun-ai bitfun-ai@users.noreply.github.com

Every cross-machine model catalog read carried the full models.dev
projections. Opening Model Settings, and creating a session, therefore
waited on a multi-MiB body that the receiver already owns: models.dev is
public data and every host refreshes its own snapshot.

The catalog that crosses a machine boundary now carries only the configured
models, the defaults and the session selection. That covers the peer host
answer, the remote session poll, the mobile and bot controllers, and the
CLI peer host. The slim build still reports the built-in provider catalog
revision, so `RemoteModelCatalog::version` does not move and a controller's
change detection is unaffected.

A controller that renders Model Settings while a peer is selected composes
the surface from the host facts plus its own snapshot, through the new
controller-local `get_local_models_dev_catalogs` command. Session creation
reads three small config keys and asks for one model's reasoning projection
instead of pulling the whole catalog.

Co-authored-by: bitfun-ai <bitfun-ai@users.noreply.github.com>
@bobleer
bobleer merged commit 59c8702 into GCWing:main Sep 20, 2026
13 checks passed
@bobleer
bobleer deleted the lwb/relay-model-catalog-bodies branch September 24, 2026 08:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant