Package per-model generation defaults and enforce reasoning budgets

[i386]

Problem

Mesh model packages do not carry the publisher-recommended request defaults for their exact model revision. The resolver therefore falls back to Mesh-wide sampler values even when a model card recommends materially different thinking and direct-mode settings.

Qwen3.5 exposes the failure mode: its template starts thinking by default, the console supplies neither a finite output limit nor a hard reasoning budget, and the Skippy sampling ABI does not expose llama.cpp's existing reasoning-budget sampler. If the model fails to emit its closing thinking marker, it can spend the remaining context in a repetitive thinking loop and never produce a visible answer.

The recommended Qwen3.5 sampler values are in the official model card, not config.json, tokenizer_config.json, GGUF metadata, or a generation_config.json. Package creation therefore needs a reviewed model-card discovery step rather than relying only on machine-readable Hugging Face files.

This is complementary to the related publisher-sidecar task, Package publisher metadata and use typed model defaults. Reuse its provenance and typed-resolution work; do not duplicate a second metadata ingestion path.

Required behavior

1. Add portable per-model request defaults to model packages

Extend the canonical model-package.json schema and both package-format readers with typed request-generation defaults. Keep these separate from artifact execution policy and speculative decoding.

The schema must be able to represent:

• common sampling values already supported by RequestDefaultsConfig, including temperature, top_p, top_k, min_p, typical_p, presence/frequency/repeat penalties, and related sampler controls;
• total output limit (max_tokens);
• reasoning enablement, format, and numeric or semantic reasoning budget;
• separate thinking and direct profiles when the publisher recommends different values;
• the selected default profile or an unambiguous selection rule based on resolved reasoning mode;
• provenance for every curated profile: official source repository, immutable revision, file/section, and URL.

Unknown values remain absent and use the fallback policy below. Do not guess values, copy defaults between model families, or execute instructions/code found in model cards. Older manifests without request defaults must remain valid.

2. Resolve defaults consistently

Resolve every field independently in this order:

1. explicit request value;
2. deployment/operator model override;
3. selected package profile;
4. Mesh global fallback.

Log or expose the resolved value and source so operators can distinguish request, deployment, package, and fallback behavior. Runtime code must not fetch or parse model cards.

When a package and operator config omit limits:

• effective_max_output = min(8192, remaining context after the prompt);
• if reasoning is enabled and the runtime recognizes thinking start/end boundaries, reasoning_budget = min(4096, floor(effective_max_output / 2));
• semantic levels resolve to low=1024, medium=4096, and high=8192 reasoning tokens, each clamped to at most half of the effective output limit; medium is the fallback level;
• 0 disables/ends thinking immediately;
• when boundaries are unavailable, skip the hard reasoning sampler and retain the finite total-output cap.

Explicit package, operator, and request limits may exceed these interactive fallbacks, subject to context and runtime validation.

3. Enforce the reasoning budget through Skippy

Wire the numeric budget through the public Skippy sampling ABI to the pinned llama.cpp reasoning-budget sampler. Template kwargs may still communicate model intent, but they are not enforcement.

The implementation must:

• carry the resolved budget and model-specific thinking boundary metadata into sampler construction;
• force a valid thinking terminator when the cap is reached, then continue generating the final answer within the total-output allowance;
• preserve direct and distributed/split inference parity;
• handle streaming and non-streaming responses;
• remain correct with native MTP/speculative decoding, including sampler state changes after rejected draft tokens;
• leave non-reasoning models and requests with reasoning disabled unaffected;
• distinguish unrestricted explicit behavior from an omitted value that receives the Mesh fallback.

4. Update the layer-package skill

Add a Generation defaults discovery section to .agents/skills/hf-layer-package-jobs/SKILL.md and keep the combined quantization/package skill consistent.

For the exact source revision being packaged, the workflow must inspect, in order:

1. generation_config.json and other official typed generation metadata when present;
2. tokenizer_config.json / the chat template for supported reasoning controls and boundary markers;
3. the official base-model README/model card;
4. official vendor documentation linked by the model card when the card defers to it.

The workflow must record exact citations and distinguish total-output guidance from a reasoning-only budget. It must capture task- or mode-specific profiles, prefer the official base model over a quantizer's copied README, and mark undocumented fields as unknown. The packaging dry run should print the proposed profiles and their provenance for review before upload.

5. Migrate existing model packages

Inventory every active package represented by the repository's package automation/catalog and every checked-in package manifest/fixture. For each source revision:

• research the official recommended thinking/direct generation settings;
• add typed profiles and provenance, or explicitly record that no publisher recommendation was found so runtime fallback is intentional;
• regenerate and validate the manifest/package identity as required;
• publish the updated package revision and update catalog references/checksums without rebuilding unchanged tensor artifacts when the package format permits it;
• run package preflight/certification and report coverage for all active packages.

Prioritize reasoning-capable families, including Qwen3/3.5/3.6/3.8, DeepSeek-R1, GLM, Nemotron, Seed-OSS, and any other package whose template can start a hidden thinking block. Non-reasoning packages should still receive official sampler/output recommendations when available.

Acceptance criteria

• Package-format tests cover typed defaults/profiles, validation, provenance, backward compatibility, and canonical serialization.
• Resolver tests prove field-by-field precedence and profile selection.
• A Qwen3.5 regression test shows that a capped thinking response exits the thinking block and has allowance for visible content.
• Live direct and split-mode Qwen3.5 checks cover thinking and direct profiles, streaming, and the fallback budget.
• Runtime tests cover missing boundaries, reasoning disabled, budget 0, explicit unrestricted behavior, small remaining contexts, and native MTP/speculative decoding.
• Diagnostics identify the selected profile and the source of each resolved limit/default.
• Documentation describes package schema, precedence, fallback formulas, and the difference between total-output and reasoning budgets.
• The migration produces an auditable inventory showing each active package, source revision, defaults source, applied profiles, and validation result.

Out of scope

• Parsing arbitrary model-card prose at serving time.
• Globally disabling thinking.
• Treating Hugging Face config.json as a standardized source of sampling or reasoning defaults.

[michaelneale]

yeah I think this is needed - but I don't think it will solve my issue, as others have ity too, not specific to mesh

[i386]

yeah I think this is needed - but I don't think it will solve my issue, as others have ity too, not specific to mesh

it would solve your issue though?