models: model capability metadata roadmap tracker

[RaresKeY]

Prerequisites

• I searched open issues and this has not already been proposed.
• I searched discussions and this is not already being debated there.
• This is a concrete, actionable proposal - not a vague feature request.

Area

API / Backend

Problem or Motivation

Odysseus now has the canonical capability foundation merged in #2739, but provider coverage, runtime evidence, durable storage, API/UI exposure, and routing remain separate incremental slices. Endpoint configuration can say an endpoint is generally llm or image, and provider model list responses can expose model IDs, but downstream consumers still need normalized evidence for what a specific served model can read, produce, support, or allocate at runtime.

Without those follow-up layers, several workflows remain fragile:

• Generic OpenAI-compatible /v1/models responses often prove availability only, not capability.
• Model names and IDs are tempting to parse, but they are not reliable capability evidence.
• Local servers and hosted providers expose different model-list shapes, and local backend APIs can differ across installed versions.
• Chat, vision, embeddings, image generation, audio, tools, reasoning, context limits, and deterministic controls are currently spread across endpoint config, provider-specific behavior, frontend catalogs, and heuristics.
• Future UI badges or routing decisions need to ask capability questions instead of relying on provider/model name guesses.

The goal remains incremental: extend the merged foundation with conservative provider and runtime evidence, then build storage, API/UI, and routing behavior on those explicit layers.

Proposed Solution

Use this issue as the parent tracker for the Model Capability Metadata roadmap.

The core direction is:

• extend the canonical capability/control vocabulary for model family, task, modalities, limits, deterministic controls, runtime allocation, source, confidence, and endpoint-scoped model identity;
• map explicit provider payload shape through reader modules, not model names;
• keep provider-reported claims, registry-derived facts, admin overrides, and probe-verified behavior distinct;
• preserve inventory even when capability is unknown;
• expose and route on normalized evidence only after the schema and storage model are stable.

Roadmap

Current status:

Order	Issue / PR	Purpose	Status
1	#4145 / #2739	Bootstrap canonical capability schema and shape-driven provider readers	Complete / merged
2	#4146 / #3031	Prototype request-side reasoning control using the capability/control vocabulary	Draft / rebase on #2739
3	#5575 / #5576	Expand native provider capability readers with conservative fallback	Open / review
4	#5628 / #5630	Normalize loaded runtime context allocation into a provider evidence shape	Draft / after #5576
5	#5193 / #5631	Consume Ollama loaded allocation in runtime prompt budgeting	Draft / after #5630
6	#4147	Add durable capability assertion and probe-result storage	Later
7	TBD	Add admin/manual smoke probes for tools, JSON/schema, vision, image, audio, and determinism	Later
8	TBD	Expose normalized capability records through /api/models	Later
9	TBD	Add model-picker capability indicators with evidence levels	Later
10	TBD	Add routing by explicit ModelRequirement and evidence level	Later
11	TBD	Add registry/admin override layering	Later

Current Next Step

Review the focused provider/runtime stack in dependency order:

1. models: expand native capability readers with conservative fallback #5575 / feat(models): expand provider-native capability readers #5576 expands provider-native catalog readers on the merged feat(models): define capability schema and readers #2739 foundation.
2. models: normalize loaded runtime context allocation evidence #5628 / feat(models): normalize runtime context allocation evidence #5630 dogfoods that adapter approach for already-fetched Ollama /api/ps runtime state, without performing network I/O or treating the value as a catalog limit.
3. Static context-window table overrides the endpoint's actual serving context — silent truncation in long agent sessions #5193 / fix(context): use Ollama loaded runtime allocation #5631 asks eligible Ollama endpoints for the loaded allocation and keeps the unique consumer diff limited to src/model_context.py and its tests.

#4146 / #3031 remains an independent reasoning-control follow-up. #4147 remains the separate durable evidence-storage slice.

Scope

In scope:

• canonical model capability metadata;
• endpoint-scoped model identity;
• provider-reported capability assertions;
• future verified probe results;
• deterministic-control metadata;
• shape-driven provider readers;
• versioned local backend metadata compatibility;
• conservative unknown-capability handling;
• capability evidence exposure through backend APIs;
• later UI display of evidence levels;
• later routing by explicit requirements.

Non-goals

• No model-name or prefix inference as authoritative capability evidence.
• No claim of complete provider coverage from the first PR.
• No model-picker UI changes in the bootstrap schema PR.
• No routing by capability until the evidence model is stable.
• No automatic paid or GPU-expensive probes.
• No sending provider-specific reasoning/tool/control fields to generic providers.
• No mixing provider-reader expansion with UI/routing changes unless a later issue explicitly scopes that.

Working Rules

• One slice per PR.
• Target dev.
• Keep provider readers shape-driven and test-backed.
• Preserve inventory: unknown capability should not hide a provider-reported model.
• Keep provider-reported, registry-derived, admin override, and probe-verified evidence distinct.
• Prefer unknown over wrong.
• Add provider support only after inspecting real API shape and adding tests.
• Keep UI badges as display hints, not routing truth.
• Use verified evidence for higher-risk routing decisions.
• Keep unsupported provider-native controls from leaking into generic request paths.

Foundation PR Scope

The merged foundation PR #2739 includes:

• canonical schema and query helpers;
• shape-driven provider readers for the tested bootstrap set: LM Studio, llama.cpp, Google AI Studio, OpenRouter, OpenAI, and Ollama;
• focused unit tests for source/confidence behavior and no-name-inference behavior;
• Google native catalog probing with pagination and no keyed curated fallback.

That foundation slice excludes:

• model-picker UI changes;
• /api/models capability response changes;
• standalone probe tooling;
• durable probe result storage;
• routing by capability;
• claims of complete capability metadata for unvalidated providers.

Acceptance Criteria

• Bare model IDs do not produce authoritative capabilities.
• OpenAI official model-list payloads normalize to identity-only, unknown capability records.
• OpenRouter records derive capabilities from explicit catalog shape.
• Google AI Studio records derive only from explicit native model fields and do not infer image/video/audio from names.
• Ollama /api/tags remains inventory-only, while /api/show.capabilities drives claimed capability records.
• llama.cpp does not treat owned_by=llamacpp as capability evidence.
• Versioned local backend readers prefer latest native metadata, parse tested common legacy shapes, and fall back to inventory-only when no capability fields are available.
• Google endpoint probing uses native paginated /v1beta/models.
• Keyed Google catalog failures do not fall back to curated local model lists.
• Focused tests cover schema normalization, reader behavior, endpoint-scoped stable IDs, and Google catalog probing behavior.
• The issue/PR language makes clear that this is bootstrap infrastructure, not complete provider coverage from day one.

Recommended Next Issues

Foundation:

feat(models): define capability schema and readers - #4145 / #2739

Provider-reader follow-up:

models: expand native capability readers with conservative fallback - #5575 / #5576

Runtime evidence adapter:

models: normalize loaded runtime context allocation evidence - #5628 / #5630

Runtime consumer:

fix(context): use Ollama loaded runtime allocation - #5193 / #5631

Durable storage:

models: add durable capability assertion and probe-result storage - #4147

Independent reasoning-control follow-up:

reasoning: align request-side reasoning control with model capabilities - #4146 / #3031

Related Runtime Consumer

• models: normalize loaded runtime context allocation evidence #5628 / feat(models): normalize runtime context allocation evidence #5630 adds the provider-neutral runtime-allocation record and pure Ollama /api/ps adapter after feat(models): expand provider-native capability readers #5576.
• Static context-window table overrides the endpoint's actual serving context — silent truncation in long agent sessions #5193 / fix(context): use Ollama loaded runtime allocation #5631 fixes the current prompt-budgeting bug by asking the provider and passing the normalized loaded allocation into src/model_context.py.
• The integration boundary is provider query -> adapter/evidence normalization -> focused runtime consumer.
• Model maximum, explicit/configured num_ctx, and loaded runtime allocation remain distinct facts with different lifetimes.
• Context Window meter measures against the advertised maximum, and history compaction is silent #5605 is a related downstream meter/compaction report; UI notification and presentation remain outside the two backend PRs.

Are you willing to implement this?

Yes - I can open focused PRs for the remaining roadmap slices.

[nsgds]

@RaresKeY this schema is exactly the layer a few reasoning requests have been missing — I'd like to build one of the deferred follow-ups on it: a reasoning on/off control keyed off CAP_REASONING.

The reason it needs the capability layer is that toggling reasoning is heavily model-dependent. What surfaced it for me was Nemotron-VL: it defaults reasoning off and only enables it via a /think message directive — it ignores enable_thinking (on the vLLM build I tested). Other families diverge: Ollama uses top-level think: false, Qwen3 uses chat_template_kwargs.enable_thinking (or /think / /no_think), gpt-oss uses reasoning_effort and can't be fully turned off, DeepSeek-R1 can't be disabled at all. So this can't be a single global flag — it has to dispatch per model family, which is exactly why deriving it from CAP_REASONING / your readers makes sense.

Once #2739 lands, I'd propose (backend, single concern): a stored per-endpoint reasoning preference (auto / on / off) threaded through model_routes and a single shared helper in llm_core that translates it into the correct native control per family, reusing your CAP_REASONING constant + deterministic-control normalizers rather than new names. This is the user-facing counterpart to #315 (a disable/hide toggle requested there on 2026-06-05). I've prototyped the per-endpoint mechanism locally (currently driving /think for Nemotron-VL), so the mechanics are proven.

I'm deliberately not proposing UI here: there's no per-model or per-endpoint setting in the admin UI today (supports_tools is storage/API only), so how per-model settings should be surfaced is its own design decision — flagging it for the roadmap rather than guessing.

[RaresKeY]

@nsgds thanks for writing this up. The concrete model examples are useful, especially Nemotron-VL using /think, Qwen3 and Ollama taking different controls, gpt-oss only minimizing reasoning through reasoning_effort, and DeepSeek-R1 not supporting a true off path. That makes the case clearly: this cannot be one global reasoning toggle.

I'd be glad to coordinate on this if you want to work on it together. Your local prototype around the per-endpoint mechanism sounds like a good starting point for validating the shape before it becomes a broader UI/API decision.

The way I am thinking about #2737 is that Odysseus owns the canonical capability layer. Provider-specific shapes are inputs that we map only as far as Odysseus needs them. In #2739 that boundary is still intentionally narrow: model_capabilities.py defines shape and normalization, and the provider readers map explicit payload fields into that shape without treating model names as authoritative capability evidence.

Your reasoning-control examples fit that same implementation pattern. The useful layering seems to be:

vendor-native behavior/control -> normalized helper/reader result -> Odysseus canonical capability/control metadata

So I agree this belongs as a follow-up on top of #2739 rather than expanding the capability-schema PR itself. The backend-only shape you outlined is the right direction:

• stored per-endpoint reasoning preference: auto / on / off;
• one shared llm_core helper that translates that preference into provider/model-family-specific request controls;
• keep the vendor-specific knob mapping behind that helper/reader boundary instead of scattering it through routes, chat flow, or UI code;
• reuse CAP_REASONING plus the source/confidence/assertion/deterministic-control vocabulary from feat(models): define capability schema and readers #2739, adding only the minimal canonical control terms we actually need;
• conservative fallback when a model cannot truly disable reasoning, without leaking unsupported fields to generic providers.

That also lines up with the scoping lesson from #968: reasoning-specific fields should only be preserved or sent for providers/models that require or advertise them.

Let's keep UI out of the first follow-up. #315 shows the user need, but the admin surface for per-endpoint/per-model settings needs a separate design decision. Once #2739 lands, a small backend PR or tracking issue for the mechanism above would be the right next step.

[nsgds]

Thanks. Works for me — I've opened a draft prototype so there's something concrete to look at: #3031. It's for validating the shape, not to merge ahead of #2739.

It sticks to the boundaries you outlined:

• Helper boundary — family detection and the /think logic are all in src/reasoning_control.py; stream_llm just calls it, and the route/CRUD only stores the auto/on/off preference. No vendor knob in routes, chat, or UI.
• No leak to generic providers — the directive is gated to the /think-dialect family, so anything else gets nothing (there's a unit test for it).
• feat(models): define capability schema and readers #2739 vocabulary — it uses a standalone reasoning_modes preference for now, but the "which family / does it reason?" lookup is isolated behind one function, so it can re-key onto CAP_REASONING and your capability vocabulary once feat(models): define capability schema and readers #2739 lands.
• Backend-only, no UI. This is the prototype; the small backend PR would follow once feat(models): define capability schema and readers #2739 merges.

Scope is Category 1 (/think) only; categories 2–5 and graded effort are documented in the helper as extension points. Open to adjusting it to fit the schema.

[RaresKeY]

Small update from #2739 / #3031 alignment:

We added a narrow #2739 schema piece for reasoning controls: canonical REASONING_CONTROL_* mechanisms and REASONING_CONTROL_VALUE_* values. The intent is still that #2739 owns capability/control evidence, source, confidence, and endpoint-scoped model identity, while #3031 owns runtime preference and request mutation.

In short:

• CAP_REASONING remains model-level capability.
• REASONING_CONTROL_* describes provider/engine control mechanism evidence, e.g. /think, think: false, enable_thinking, budgets, effort, etc.
• REASONING_CONTROL_VALUE_* describes provider-supported values: on, off, auto. Here auto means the provider explicitly supports adaptive/dynamic/vendor-decided reasoning.

Longer alignment note posted on #3031:
https://github.com/pewdiepie-archdaemon/odysseus/pull/3031#issuecomment-4637223138