Skip to content

Support litellm or langgraph as alternative LLM backends #33

Description

@benthomasson

Problem

The LLM integration is currently hardcoded to invoke the claude CLI via subprocess. This works but limits model portability — switching models requires a CLI tool that accepts the same stdin/stdout contract.

Proposed solution

Add support for alternative LLM backends behind the existing MODEL_COMMANDS registry:

  • litellm — unified Python API for 100+ LLM providers (OpenAI, Anthropic, Gemini, local models)
  • langgraph — for users who want agent-style orchestration

This could be implemented as:

  1. A --backend flag (cli, litellm, langgraph) on pipeline/summarize/propose/exam commands
  2. Refactor llm.py:invoke_sync to dispatch to the selected backend
  3. Keep the CLI subprocess path as the default for backward compatibility

Context

Related to #17 (stringly-typed-llm-integration). The current subprocess-based architecture makes model swapping look trivial but prompt templates may embed implicit model-specific format expectations. A Python SDK backend would enable structured output parsing and better error handling.

Workaround

Currently users can add entries to MODEL_COMMANDS for any CLI tool that accepts prompts on stdin and returns completions on stdout.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions