Skip to content

Typed video input support: VideoURL multimodal class #275

Description

@Eigenwise

PR #274 adds video input support by detecting raw {"type": "video_url", "video_url": {"url": ...}} dicts in schema fields. The use case is real (MiniMax-M3 and other OpenAI-compatible providers accept video_url content parts), but the implementation has a blocker: LiteLLM's token_counter raises on video_url parts, so any agent with max_context_tokens set crashes on every run() once a video is in history. Shape-detection of bare dicts also cuts against the schema-first design. Details in the review on #274.

Plan:

  • Add a typed VideoURL class (url, optional fps / detail, to_openai() emitting the OpenAI-compatible content part), exported from the package root
  • Treat it as a first-class multimodal type in ChatHistory._extract_multimodal_info, same as Instructor's Image / Audio / PDF
  • get_history() emits the content-part dict (Instructor passes plain dicts through to the provider unchanged, verified against 1.14.x)
  • Token counting uses a text placeholder for video parts since LiteLLM can't count them
  • dump() / load() round-trips it natively as a plain Pydantic model

Instructor has no Video type of its own yet; upstream ask is 567-labs/instructor#2520. If that lands we can swap it in behind the same seam.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions