Conversation
stikves
force-pushed
the
sukru/vlm-consolidation-b3b4
branch
from
September 14, 2026 20:34
b065d86 to
9e90ffc
Compare
stikves
force-pushed
the
sukru/vlm-consolidation-c1c2
branch
from
September 14, 2026 20:34
e947c89 to
2891131
Compare
stikves
force-pushed
the
sukru/vlm-consolidation-b3b4
branch
from
September 14, 2026 21:09
9e90ffc to
51d4c22
Compare
stikves
force-pushed
the
sukru/vlm-consolidation-c1c2
branch
from
September 14, 2026 21:10
2891131 to
67d3be5
Compare
…d prefill (B1+B2) B1: Replace the VLM engine's hand-rolled KV cache (keyCache/valueCache/currentKVCapacity plus ensureKVCapacity/copyCache/zeroFill) with the shared StateHandlerFactory and runWithStates path used by the text engine. Growth semantics unchanged (initial 256, 2x). Restore the per-realloc "KV cache grew" log in GrowingNDArrayState. Read back hasNonTruncatableStates and guard reset(to:) against partial reset for recurrent-state models, matching the text engine. B2: Add GenerationTokenBox for the active-token busy/cancel/install bookkeeping both engines duplicated, and runChunkedPrefill() to replace both processChunkedPrompt implementations. The text engine passes a non-zero heldBack and routes non-held-back chunks through the prefill graph; the VLM engine passes heldBack = 0. Adds GenerationTokenBox and chunked-prefill unit tests.
stikves
force-pushed
the
sukru/vlm-consolidation-b3b4
branch
from
September 14, 2026 21:26
51d4c22 to
9f083c2
Compare
stikves
force-pushed
the
sukru/vlm-consolidation-c1c2
branch
from
September 14, 2026 21:26
67d3be5 to
7c19bcc
Compare
Both sequential engines' generation iterators computed the maxTokens clamp and the forced-vs-sampled next token with identical logic. Extract them into a SequentialIterator helper namespace (clampMaxTokens, nextToken) that both call. Kept intentionally minimal and pure value-semantics: each engine retains its own next() control flow (the text engine's batched forced-continuation PPL path and prefix-cache bookkeeping; the VLM's image prefill), so the shared helpers don't touch the engines' lifetime/MutableViews machinery. Behavior-preserving: the clamp arithmetic and the copy-on-write sampling semantics are unchanged.
With KV, token box, chunked prefill, and iterator helpers now shared with CoreAISequentialEngine, remove the '// TODO: Refactor to re-use common components' marker and update the class doc to describe the shared decode-loop machinery. Fold the repeated processedTokenCount/kvCache.reset/additionalStates zeroing (reset and warmup) into a private clearGenerationState() helper. The pipelined-variant TODO stays — that remains a separate future effort.
Add EngineFactory.createEngine(bundle:options:), a multi-component entry point alongside the single-asset createEngine(config:modelURL:options:). VLM bundles (kind == .vlm) carry three assets (vision, embedding, main) that the single-asset API can't express; they route to the sequential VLM engine, selected by bundle kind rather than variant auto-detection (a VLM main graph is indistinguishable from a plain dynamic LLM at the graph level). Other kinds resolve their main asset and fall through to the single-asset path. The factory now owns VLM config assembly (makeVLMConfig) and applies chunking overrides to the base config in one place, mirroring selectEngine for the text path. Sequential component preparation matches the runner's existing choice. Adds unit tests for config assembly and override application (asset-free).
Migrate both VLM construction sites to EngineFactory.createEngine(bundle:options:): - LLMRunnerMain: replace the if-isVLM/else fork with a single factory call. - CoreAIVisionLanguageModel (FM adapter): call the factory and downcast to the concrete engine for the executor. This also moves the adapter's three component preparations from concurrent to sequential, matching the runner (concurrent preparation could trip Core AI specialization). - CoreAISequentialVLMEngine.init: drop the resolvedBase chunking-override block; the factory (makeVLMConfig) now applies overrides before construction, so the engine takes an already-resolved config and no longer copies/rebuilds it. No caller constructs CoreAISequentialVLMEngine directly anymore; chunking-override application lives in exactly one place (the factory).
stikves
force-pushed
the
sukru/vlm-consolidation-b3b4
branch
from
September 14, 2026 21:30
9f083c2 to
8d200cb
Compare
stikves
force-pushed
the
sukru/vlm-consolidation-c1c2
branch
from
September 14, 2026 21:30
7c19bcc to
927341a
Compare
stikves
force-pushed
the
sukru/vlm-consolidation-b3b4
branch
2 times, most recently
from
September 15, 2026 01:52
3d6e3f3 to
b50e30a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Final step of the VLM sequential-engine consolidation (steps C1 and C2). Stacked on #3. Routes VLM
construction through
EngineFactoryand removes the CLIif isVLM { ... } else { ... }constructionfork, the review-feedback item that motivated this work.
Targets
sukru/vlm-consolidation-b3b4so the diff is limited to the C1/C2 changes. Will beretargeted up the stack as the lower PRs merge.
Changes
C1: Add
EngineFactory.createEngine(bundle:options:), routing by bundle kind (.vlmgoes to aprivate
makeVLMEngine). Routing by kind avoids variant auto-detection, which cannot distinguish aVLM
maingraph from a plain dynamic LLM. Non-VLM kinds resolvemainand use the existingsingle-asset path. The factory owns
makeVLMConfig(base config plus chunking overrides) andprepares the three VLM components sequentially, since concurrent preparation can trip Core AI
specialization. Adds asset-free unit tests (
EngineFactoryVLMTests).C2: Migrate call sites and delete the fork.
LLMRunnerMainreplaces its construction fork with asingle
createEngine(bundle:)call. The FoundationModels adapter (CoreAIVisionLanguageModel)routes through the factory, which also removes a latent bug where it prepared the three components
concurrently.
CoreAISequentialVLMEngine.initdrops itsresolvedBaseconfig-rebuild block, sincethe factory applies overrides before construction.
Validation
Bit-exact against the pre-refactor baseline at this tip: text greedy 128/128, T2 PPL exact, VLM
greedy 115/115 and 400/400.
xcodebuild buildandxcodebuild testpass;EngineFactoryVLMTestsand the B1/B2 unit suites aregreen.