Expose nUbatch independently from nBatch (+ surface ctx.nBatch/nUbatch)

[lloyal-research]

Summary

nBatch currently drives both llama_context_params.n_batch and n_ubatch, so n_ubatch cannot be tuned independently. Expose a separate nUbatch option (default = nBatch), and surface the configured nBatch/nUbatch from SessionContext so higher layers stop guessing at them.

Current wiring (verified)

• One option sets both physical and logical batch:
  • src/SessionContext.cpp:1683-1687 parses nBatch (default N_BATCH_INIT = 512, liblloyal/include/lloyal/common.hpp:71).
  • src/SessionContext.cpp:1775-1776: ctx_params.n_batch = ctx_params.n_ubatch = nBatch;
• The SDK type documents this coupling explicitly: packages/sdk/src/types.ts:193 ("Also sets llama_context_params.n_batch and n_ubatch"), :196 nBatch?: number — there is no nUbatch field.
• SessionContext retains only _nBatch (src/SessionContext.hpp:296) and exposes no batch accessor — only vocabSize / memorySize (src/SessionContext.cpp:853-854).

Consequence: since n_ubatch == n_batch and any submitted native batch is ≤ n_batch, each llama_decode() currently resolves to a single physical ubatch. Tuning n_ubatch (e.g. for pipeline-parallel microbatch overlap on multi-GPU) is not reachable through the API. Note this is a hypothesis about unlocking overlap — it needs measurement, not an assumed win (see caveat below).

The packing boundary is already correct — don't move it

BranchStore should keep building the largest useful logical batch up to n_batch and let llama.cpp split it into n_ubatch-sized physical microbatches internally. It already does exactly this and already reads the value dynamically:

• liblloyal/include/lloyal/branch.hpp:1011: const int32_t batch_limit = static_cast<int32_t>(llama_n_batch(ctx));
• :1019: auto chunks = decode::bin_pack(spans.data(), n, batch_limit);
• decode_each() submits one row per live branch in a single llama_decode (:374, :921).

BranchStore already receives the context via init_tenancy(ctx) (branch.hpp:499), so it can read llama_n_batch(ctx) / llama_n_ubatch(ctx) for observability without any new parameter. No packing/chunk-boundary change.

Both accessors exist upstream: llama.h:542-543 (llama_n_batch, llama_n_ubatch).

Proposed change

1. nUbatch context option — parse it alongside nBatch (same pattern as SessionContext.cpp:1683-1687), default = nBatch, constraint 1 ≤ nUbatch ≤ nBatch, then:

   ctx_params.n_batch  = static_cast<uint32_t>(nBatch);
   ctx_params.n_ubatch = static_cast<uint32_t>(nUbatch);

   Preserves current behaviour unless explicitly tuned.
2. SDK type — add nUbatch?: number to ContextOptions (packages/sdk/src/types.ts), documented as "physical microbatch, defaults to nBatch".
3. Expose ctx.nBatch / ctx.nUbatch from SessionContext (InstanceAccessors backed by llama_n_batch/llama_n_ubatch), for config reporting / benchmark traces / assertions.

Latent correctness bug this fixes (not just cosmetics)

packages/agents (lloyal-sdk) hardcodes ContextPressure.ASSUMED_N_BATCH = 512 because the binding doesn't expose the real value — with an explicit TODO to switch to ctx.nBatch once available:

• agent-pool.ts:151 static readonly ASSUMED_N_BATCH = 512; (+ TODO :148-149)
• agent-pool.ts:823 validates hardLimit < nBatch against that hardcoded 512.

If a deployment sets nBatch > 512, the recovery-reserve invariant validates against the wrong (smaller) number and can pass a hardLimit that is actually below the real batch size — i.e. the exact next-batch OOM the invariant exists to prevent. Exposing ctx.nBatch closes that hole.

Already a live non-512 case: the reranker

packages/rig/src/reranker.ts:51 derives nBatch = opts?.nBatch ?? Math.floor(nCtx / nSeqMax). With the current defaults (nSeqMax = 10, reranker.ts:49; nCtx = 4096) this is floor(4096/10) = 409 — so the reranker context already runs n_batch = n_ubatch = 409, not 512. That both (a) confirms nBatch is already deployment-derived and (b) is a second reason the ASSUMED_N_BATCH = 512 constant is wrong for at least one shipped configuration.

Caveat for benchmarking (hybrid/SSM)

The default production model is a gated DeltaNet hybrid. llama.h:545 exposes llama_n_rs_seq — recurrent-state sequences are a distinct axis, and llama.cpp's ubatch splitting for recurrent/hybrid architectures carries equal-split constraints. Small-nUbatch behaviour on the hybrid should be measured independently (decode cohort vs prefill cohort, per-arch), not assumed to mirror a pure transformer. nUbatch is best treated as a runtime-host / model-residency configuration, not an agent-pool option; keep the default = nBatch and tune per deployment.