Add support for multimodal input

[noorbhatia]

Thanks a lot for making this package, I am keen to switching to this for a production app.

Many of the language model providers that AnyLanguageModel already supports have strong multimodal capabilities:

MLX: Supports vision-language models like Qwen2-VL, and other multimodal models
OpenAI: GPT-4o, GPT-4o-mini, and GPT-4-turbo all support vision

Currently, users cannot leverage these vision capabilities through AnyLanguageModel, which limits the library's usefulness for multimodal applications.

Use Cases:

• Image captioning and description
• OCR and document understanding

[mattt]

Hi @noorbhatia. I agree that the Foundation Models framework's lack of support for multi-modality limits its usefulness. As an API compatibility layer, AnyLanguageModel is inherently a lowest common denominator; it's too bad that the limitations happen at Apple's API.

I suspect that Apple will update Foundation Models to support sending images with prompts in iOS 27. But I don't want to wait that long. I'm keen to support the extent of what's already possible with other language backends.

Kind of like how the Game of Thrones TV series ran past its source material while waiting for the next book to be written. There's a risk we write something that breaks the eventual canon — but hey, that's what deprecation warnings are for.

(Or maybe it's more like Borges, writing stories critiquing books that don't exist)

So let's put on our best Cupertino accent and guess at what that eventual API should be. Could it be as simple as adding a new case to Transcript.Segment, .image(ImageSegment)?

[mattt]

@noorbhatia I just opened up #21 with an initial implementation. If you have a moment, please give the mattt/image-content branch a try. I've tested with OpenAI and Anthropic adapters so far (I'm away from my main machine, and can't easily run on-device models)

[noorbhatia]

Thanks a lot for the initial implementation @mattt , I'll get back with updates!

[noorbhatia]

Hi @mattt I got it working for MLX models, tested with Qwen2-VL-2B-Instruct-4bit. How do I go about raising a PR?

[mattt]

@noorbhatia That's great to hear! I recently force-pushed a fix to the build errors when building for MLX. If you have any other changes to put on that, please open a PR against mattt/image-content or feel free to drop a diff as a comment in #21 and I can patch that myself.

[noorbhatia]

Hi @mattt , I had opened a PR #23

[mattt]

@noorbhatia Thanks again for your help with this! I just merged #21 and cut 0.4.0. I look forward to implementing image support for CoreML and llama.cpp models (and fixing any bugs we find in MLX support) in follow-up PRs soon.