PR #274 adds video input support by detecting raw {"type": "video_url", "video_url": {"url": ...}} dicts in schema fields. The use case is real (MiniMax-M3 and other OpenAI-compatible providers accept video_url content parts), but the implementation has a blocker: LiteLLM's token_counter raises on video_url parts, so any agent with max_context_tokens set crashes on every run() once a video is in history. Shape-detection of bare dicts also cuts against the schema-first design. Details in the review on #274.
Plan:
- Add a typed
VideoURL class (url, optional fps / detail, to_openai() emitting the OpenAI-compatible content part), exported from the package root
- Treat it as a first-class multimodal type in
ChatHistory._extract_multimodal_info, same as Instructor's Image / Audio / PDF
get_history() emits the content-part dict (Instructor passes plain dicts through to the provider unchanged, verified against 1.14.x)
- Token counting uses a text placeholder for video parts since LiteLLM can't count them
dump() / load() round-trips it natively as a plain Pydantic model
Instructor has no Video type of its own yet; upstream ask is 567-labs/instructor#2520. If that lands we can swap it in behind the same seam.
PR #274 adds video input support by detecting raw
{"type": "video_url", "video_url": {"url": ...}}dicts in schema fields. The use case is real (MiniMax-M3 and other OpenAI-compatible providers acceptvideo_urlcontent parts), but the implementation has a blocker: LiteLLM'stoken_counterraises onvideo_urlparts, so any agent withmax_context_tokensset crashes on everyrun()once a video is in history. Shape-detection of bare dicts also cuts against the schema-first design. Details in the review on #274.Plan:
VideoURLclass (url, optionalfps/detail,to_openai()emitting the OpenAI-compatible content part), exported from the package rootChatHistory._extract_multimodal_info, same as Instructor'sImage/Audio/PDFget_history()emits the content-part dict (Instructor passes plain dicts through to the provider unchanged, verified against 1.14.x)dump()/load()round-trips it natively as a plain Pydantic modelInstructor has no
Videotype of its own yet; upstream ask is 567-labs/instructor#2520. If that lands we can swap it in behind the same seam.