diff --git a/cookbook/Benchmarking_LLMs_by_use_case.ipynb b/cookbook/Benchmarking_LLMs_by_use_case.ipynb
new file mode 100644
index 00000000..6ea6211b
--- /dev/null
+++ b/cookbook/Benchmarking_LLMs_by_use_case.ipynb
@@ -0,0 +1,753 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4Cq-_Y-TKf0r"
+ },
+ "source": [
+ "# LiteLLM - Benchmark Llama2, Claude1.2 and GPT3.5 for a use case\n",
+ "In this notebook for a given use case we run the same question and view:\n",
+ "* LLM Response\n",
+ "* Response Time\n",
+ "* Response Cost\n",
+ "\n",
+ "## Sample output for a question\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "O3ENsWYB27Mb"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install litellm"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Pk55Mjq_3DiR"
+ },
+ "source": [
+ "## Example Use Case 1 - Code Generator\n",
+ "### For this use case enter your system prompt and questions\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "metadata": {
+ "id": "_1SZYJFB3HmQ"
+ },
+ "outputs": [],
+ "source": [
+ "# enter your system prompt if you have one\n",
+ "system_prompt = \"\"\"\n",
+ "You are a coding assistant helping users using litellm.\n",
+ "litellm is a light package to simplify calling OpenAI, Azure, Cohere, Anthropic, Huggingface API Endpoints\n",
+ "--\n",
+ "Sample Usage:\n",
+ "```\n",
+ "pip install litellm\n",
+ "from litellm import completion\n",
+ "## set ENV variables\n",
+ "os.environ[\"OPENAI_API_KEY\"] = \"openai key\"\n",
+ "os.environ[\"COHERE_API_KEY\"] = \"cohere key\"\n",
+ "messages = [{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}]\n",
+ "# openai call\n",
+ "response = completion(model=\"gpt-3.5-turbo\", messages=messages)\n",
+ "# cohere call\n",
+ "response = completion(\"command-nightly\", messages)\n",
+ "```\n",
+ "\n",
+ "\"\"\"\n",
+ "\n",
+ "\n",
+ "# qustions/logs you want to run the LLM on\n",
+ "questions = [\n",
+ " \"what is litellm?\",\n",
+ " \"why should I use LiteLLM\",\n",
+ " \"does litellm support Anthropic LLMs\",\n",
+ " \"write code to make a litellm completion call\",\n",
+ "]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "AHH3cqeU3_ZT"
+ },
+ "source": [
+ "## Running questions\n",
+ "### Select from 100+ LLMs here: https://docs.litellm.ai/docs/providers"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "BpQD4A5339L3"
+ },
+ "outputs": [],
+ "source": [
+ "from litellm import completion, completion_cost\n",
+ "import os\n",
+ "import time\n",
+ "\n",
+ "# optional use litellm dashboard to view logs\n",
+ "# litellm.use_client = True\n",
+ "# litellm.token = \"ishaan_2@berri.ai\" # set your email\n",
+ "\n",
+ "\n",
+ "# set API keys\n",
+ "os.environ['TOGETHERAI_API_KEY'] = \"\"\n",
+ "os.environ['OPENAI_API_KEY'] = \"\"\n",
+ "os.environ['ANTHROPIC_API_KEY'] = \"\"\n",
+ "\n",
+ "\n",
+ "# select LLMs to benchmark\n",
+ "# using https://api.together.xyz/playground for llama2\n",
+ "# try any supported LLM here: https://docs.litellm.ai/docs/providers\n",
+ "\n",
+ "models = ['togethercomputer/llama-2-70b-chat', 'gpt-3.5-turbo', 'claude-instant-1.2']\n",
+ "data = []\n",
+ "\n",
+ "for question in questions: # group by question\n",
+ " for model in models:\n",
+ " print(f\"running question: {question} for model: {model}\")\n",
+ " start_time = time.time()\n",
+ " # show response, response time, cost for each question\n",
+ " response = completion(\n",
+ " model=model,\n",
+ " max_tokens=500,\n",
+ " messages = [\n",
+ " {\n",
+ " \"role\": \"system\", \"content\": system_prompt\n",
+ " },\n",
+ " {\n",
+ " \"role\": \"user\", \"content\": question\n",
+ " }\n",
+ " ],\n",
+ " )\n",
+ " end = time.time()\n",
+ " total_time = end-start_time # response time\n",
+ " # print(response)\n",
+ " cost = completion_cost(response) # cost for completion\n",
+ " raw_response = response['choices'][0]['message']['content'] # response string\n",
+ "\n",
+ "\n",
+ " # add log to pandas df\n",
+ " data.append(\n",
+ " {\n",
+ " 'Model': model,\n",
+ " 'Question': question,\n",
+ " 'Response': raw_response,\n",
+ " 'ResponseTime': total_time,\n",
+ " 'Cost': cost\n",
+ " })"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "apOSV3PBLa5Y"
+ },
+ "source": [
+ "## View Benchmarks for LLMs"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1000
+ },
+ "id": "CJqBlqUh_8Ws",
+ "outputId": "e02c3427-d8c6-4614-ff07-6aab64247ff6"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Question: does litellm support Anthropic LLMs\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "
Model
\n",
+ "
Question
\n",
+ "
Response
\n",
+ "
ResponseTime
\n",
+ "
Cost
\n",
+ "
\n",
+ " \n",
+ " \n",
+ "
\n",
+ "
6
\n",
+ "
togethercomputer/llama-2-70b-chat
\n",
+ "
does litellm support Anthropic LLMs
\n",
+ "
Yes, litellm supports Anthropic LLMs.\\n\\nIn the example usage you provided, the `completion` function is called with the `model` parameter set to `\"gpt-3.5-turbo\"` for OpenAI and `\"command-nightly\"` for Cohere.\\n\\nTo use an Anthropic LLM with litellm, you would set the `model` parameter to the name of the Anthropic model you want to use, followed by the version number, if applicable. For example:\\n```\\nresponse = completion(model=\"anthropic-gpt-2\", messages=messages)\\n```\\nThis would call the Anthropic GPT-2 model to generate a completion for the given input messages.\\n\\nNote that you will need to set the `ANTHROPIC_API_KEY` environment variable to your Anthropic API key before making the call. You can do this by running the following command in your terminal:\\n```\\nos.environ[\"ANTHROPIC_API_KEY\"] = \"your-anthropic-api-key\"\\n```\\nReplace `\"your-anthropic-api-key\"` with your actual Anthropic API key.\\n\\nOnce you've set the environment variable, you can use the `completion` function with the `model` parameter set to an Anthropic model name to call the Anthropic API and generate a completion.
\n",
+ "
21.513009
\n",
+ "
0.001347
\n",
+ "
\n",
+ "
\n",
+ "
7
\n",
+ "
gpt-3.5-turbo
\n",
+ "
does litellm support Anthropic LLMs
\n",
+ "
No, currently litellm does not support Anthropic LLMs. It mainly focuses on simplifying the usage of OpenAI, Azure, Cohere, and Huggingface API endpoints.
\n",
+ "
8.656510
\n",
+ "
0.000342
\n",
+ "
\n",
+ "
\n",
+ "
8
\n",
+ "
claude-instant-1.2
\n",
+ "
does litellm support Anthropic LLMs
\n",
+ "
Yes, litellm supports calling Anthropic LLMs through the completion function.\\n\\nTo use an Anthropic model with litellm:\\n\\n1. Set the ANTHROPIC_API_KEY environment variable with your Anthropic API key\\n\\n2. Pass the model name as the 'model' argument to completion(). Anthropic model names follow the format 'anthropic/<model_name>'\\n\\nFor example:\\n\\n```python \\nimport os\\nfrom litellm import completion\\n\\nos.environ[\"ANTHROPIC_API_KEY\"] = \"your_anthropic_api_key\"\\n\\nmessages = [{\"content\": \"Hello\", \"role\": \"user\"}]\\n\\nresponse = completion(model=\"anthropic/constitutional\", messages=messages)\\n```\\n\\nThis would call the Constitutional AI model from Anthropic.\\n\\nSo in summary, litellm provides a simple interface to call any Anthropic models as long as you specify the model name correctly and set the ANTHROPIC_API_KEY env variable.
Litellm is a lightweight Python package that simplifies calling various AI API endpoints, including OpenAI, Azure, Cohere, Anthropic, and Hugging Face. It provides a convenient interface for making requests to these APIs, allowing developers to easily integrate them into their applications. With Litellm, developers can quickly and easily interact with multiple AI models and services, without having to handle the details of authentication, API calls, and response parsing. This makes it easier to build and deploy AI-powered applications, and can help developers save time and effort.
\n",
+ "
13.479644
\n",
+ "
0.000870
\n",
+ "
\n",
+ "
\n",
+ "
1
\n",
+ "
gpt-3.5-turbo
\n",
+ "
what is litellm?
\n",
+ "
litellm is a light package that provides a simplified interface for making API calls to various language models and APIs. It abstracts away the complexities of handling network requests, authentication, and response parsing, making it easier for developers to integrate powerful language models into their applications.\\n\\nWith litellm, you can quickly make API calls to models like OpenAI's GPT-3.5 Turbo, Azure's Text Analytics, Cohere's Command API, Anthropic's API, and Huggingface's models. It also supports additional functionality like conversational AI, summarization, translation, and more.\\n\\nBy using litellm, you can focus on your application logic without getting tangled in the details of API integration, allowing you to quickly build intelligent and conversational applications.
\n",
+ "
8.324332
\n",
+ "
0.000566
\n",
+ "
\n",
+ "
\n",
+ "
2
\n",
+ "
claude-instant-1.2
\n",
+ "
what is litellm?
\n",
+ "
litellm is a Python library that simplifies calling various AI API endpoints like OpenAI, Azure, Cohere, Anthropic, and Huggingface. \\n\\nSome key things to know about litellm:\\n\\n- It provides a consistent interface for completing prompts and generating responses from different AI models through a single method called completion().\\n\\n- You specify the API (e.g. OpenAI, Cohere etc.) and model either by name or by setting environment variables before making the completion call.\\n\\n- This avoids having to use different SDKs or APIs for each provider and standardizes the call structure. \\n\\n- It handles things like setting headers, encoding inputs, parsing responses so the user doesn't have to deal with those details.\\n\\n- The goal is to make it easy to try different AI APIs and models without having to change code or learn different interfaces.\\n\\n- It's lightweight with no other dependencies required besides what's needed for each API (e.g. openai, azure SDKs etc.).\\n\\nSo in summary, litellm is a small library that provides a common way to interact with multiple conversational AI APIs through a single Python method, avoiding the need to directly use each provider's specific SDK.
\\nThere are several reasons why you might want to use LiteLLM:\\n\\n1. Simplified API calls: LiteLLM provides a simple and consistent API for calling various language models, making it easier to use multiple models and switch between them.\\n2. Environment variable configuration: LiteLLM allows you to set environment variables for API keys and model names, making it easier to manage and switch between different models and APIs.\\n3. Support for multiple models and APIs: LiteLLM supports a wide range of language models and APIs, including OpenAI, Azure, Cohere, Anthropic, and Hugging Face.\\n4. Easy integration with popular frameworks: LiteLLM can be easily integrated with popular frameworks such as PyTorch and TensorFlow, making it easy to use with your existing codebase.\\n5. Lightweight: LiteLLM is a lightweight package, making it easy to install and use, even on resource-constrained devices.\\n6. Flexible: LiteLLM allows you to define your own models and APIs, making it easy to use with custom models and APIs.\\n7. Extensive documentation: LiteLLM has extensive documentation, making it easy to get started and learn how to use the package.\\n8. Active community: LiteLLM has an active community of developers and users, making it easy to get help and feedback on your projects.\\n\\nOverall, LiteLLM can help you to simplify your workflow, improve your productivity, and make it easier to work with multiple language models and APIs.
\n",
+ "
23.777885
\n",
+ "
0.001443
\n",
+ "
\n",
+ "
\n",
+ "
4
\n",
+ "
gpt-3.5-turbo
\n",
+ "
why should I use LiteLLM
\n",
+ "
LiteLLM is a lightweight Python package that simplifies the process of making API calls to various language models. Here are some reasons why you should use LiteLLM:\\n\\n1. **Simplified API Calls**: LiteLLM abstracts away the complexity of making API calls to different language models. It provides a unified interface for invoking models from OpenAI, Azure, Cohere, Anthropic, Huggingface, and more.\\n\\n2. **Easy Integration**: LiteLLM seamlessly integrates with your existing codebase. You can import the package and start making API calls with just a few lines of code.\\n\\n3. **Flexibility**: LiteLLM supports a variety of language models, including GPT-3, GPT-Neo, chatGPT, and more. You can choose the model that suits your requirements and easily switch between them.\\n\\n4. **Convenience**: LiteLLM handles the authentication and connection details for you. You just need to set the relevant environment variables, and the package takes care of the rest.\\n\\n5. **Quick Prototyping**: LiteLLM is ideal for rapid prototyping and experimentation. With its simple API, you can quickly generate text, chat with models, and build interactive applications.\\n\\n6. **Community Support**: LiteLLM is actively maintained and supported by a community of developers. You can find help, share ideas, and collaborate with others to enhance your projects.\\n\\nOverall, LiteLLM simplifies the process of making API calls to language models, saving you time and effort while providing flexibility and convenience.
\n",
+ "
12.109881
\n",
+ "
0.000881
\n",
+ "
\n",
+ "
\n",
+ "
5
\n",
+ "
claude-instant-1.2
\n",
+ "
why should I use LiteLLM
\n",
+ "
Here are some key reasons why you may want to consider using LiteLLM:\\n\\n- Simplifies calling multiple large language models - LiteLLM provides a unified API to call models from different providers like OpenAI, Azure, HuggingFace, Anthropic etc. This avoids having to deal with different APIs from each provider.\\n\\n- Easy to get started - LiteLLM is very lightweight and simple to install with just one pip install. It has minimal configuration needed with just API keys. This lowers the barrier to start experimenting with LLMs.\\n\\n- Standard interface - The completion API provides a standard way to send messages and get responses back regardless of underlying model. This keeps the code interface consistent. \\n\\n- Model agnostic - Your code doesn't need to change to switch between different models. You can easily compare performance of models from different providers.\\n\\n- Open source - As it is open source, the library can be expanded to support more models and providers over time via community contributions.\\n\\n- Cost effective - LiteLLM helps avoid vendor lock-in. You can optimize costs by testing multiple models and choosing the best suitable and cost-effective option for your needs.\\n\\nSo in summary, L
\\nTo make a completion call using litellm, you can use the `completion` function from the litellm package. Here's an example of how to use it:\\n```\\nfrom litellm import completion\\n\\n# Set ENV variables\\nos.environ[\"OPENAI_API_KEY\"] = \"your_openai_api_key\"\\nos.environ[\"COHERE_API_KEY\"] = \"your_cohere_api_key\"\\n\\n# Define the messages to be completed\\nmessages = [\\n {\\n \"content\": \"Hello, how are you?\",\\n \"role\": \"user\"\\n }\\n]\\n\\n# Make a completion call using OpenAI\\nresponse = completion(model=\"gpt-3.5-turbo\", messages=messages)\\n\\n# Make a completion call using Cohere\\nresponse = completion(\"command-nightly\", messages)\\n```\\nIn this example, we first set the ENV variables for the OpenAI and Cohere API keys. Then, we define a list of messages to be completed, which in this case contains a single message with the content \"Hello, how are you?\" and the role \"user\".\\n\\nNext, we make two completion calls using the `completion` function from litellm. The first call uses the OpenAI model `gpt-3.5-turbo` and passes in the list of messages. The second call uses the Cohere model `command-nightly` and passes in the same list of messages.\\n\\nThe `completion` function returns a response object that contains the completed messages. You can then use the `response.messages` attribute to access the completed messages.\\n\\nHere's an example of how to access the completed messages:\\n```\\n# Print the completed messages\\nprint(response.messages)\\n```\\nThis will print the completed messages, which should look something like this:\\n```\\n[{'content': 'Hello, how are you?', 'role': 'user', 'completion': 'I am doing well, thanks for asking.'}]\\n```\\nNote that the `completion` attribute contains the completed message. You can use this attribute to retrieve the completed message and use it in your application.
\n",
+ "
32.192453
\n",
+ "
0.001821
\n",
+ "
\n",
+ "
\n",
+ "
10
\n",
+ "
gpt-3.5-turbo
\n",
+ "
write code to make a litellm completion call
\n",
+ "
Sure! Here's an example code to make a completion call using the litellm package:\\n\\n```python\\nfrom litellm import completion\\n\\n# Set the model and messages\\nmodel = \"gpt-3.5-turbo\"\\nmessages = [\\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\\n {\"role\": \"user\", \"content\": \"Who won the world series in 2020?\"},\\n {\"role\": \"assistant\", \"content\": \"The Los Angeles Dodgers won the World Series in 2020.\"},\\n {\"role\": \"user\", \"content\": \"Where was it played?\"}\\n]\\n\\n# Make the completion call\\nresponse = completion(model=model, messages=messages)\\n\\n# Print the assistant's reply\\nassistant_reply = response[\"choices\"][0][\"message\"][\"content\"]\\nprint(\"Assistant: \", assistant_reply)\\n```\\n\\nMake sure you have the litellm package installed (`pip install litellm`) and set the necessary environment variables for the API keys before running this code.
\n",
+ "
9.377155
\n",
+ "
0.000686
\n",
+ "
\n",
+ "
\n",
+ "
11
\n",
+ "
claude-instant-1.2
\n",
+ "
write code to make a litellm completion call
\n",
+ "
Here is an example of making a completion call using litellm:\\n\\n```python\\nimport os\\nfrom litellm import completion\\n\\n# Set API keys as environment variables\\nos.environ[\"OPENAI_API_KEY\"] = \"your openai api key\" \\n\\n# Conversation context \\nmessages = [{\\n \"content\": \"Hello, how can I help you today?\",\\n \"role\": \"assistant\"\\n}]\\n\\n# Make completion call with GPT-3 model\\nresponse = completion(\\n model=\"gpt-3.5-turbo\", \\n messages=messages\\n)\\n\\nprint(response)\\n```\\n\\nTo break it down:\\n\\n- Import completion from litellm\\n- Set the OPENAI_API_KEY env var \\n- Define a messages list with the conversation context\\n- Call completion(), specifying the model (\"gpt-3.5-turbo\") and messages\\n- It will return the response from the API\\n- Print the response\\n\\nThis makes a simple completion call to OpenAI GPT-3 using litellm to handle the API details. You can also call other models like Cohere or Anthropic by specifying their name instead of the OpenAI
\n",
+ "
9.839988
\n",
+ "
0.001578
\n",
+ "
\n",
+ " \n",
+ "
"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "execution_count": 22,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from IPython.core.interactiveshell import InteractiveShell\n",
+ "InteractiveShell.ast_node_interactivity = \"all\"\n",
+ "from IPython.display import HTML\n",
+ "import pandas as pd\n",
+ "\n",
+ "df = pd.DataFrame(data)\n",
+ "grouped_by_question = df.groupby('Question')\n",
+ "\n",
+ "for question, group_data in grouped_by_question:\n",
+ " print(f\"Question: {question}\")\n",
+ " HTML(group_data.to_html())\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "bmtAbC1rGVAm"
+ },
+ "source": [
+ "## Use Case 2 - Rewrite user input concisely"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "metadata": {
+ "id": "boiHO1PhGXSL"
+ },
+ "outputs": [],
+ "source": [
+ "# enter your system prompt if you have one\n",
+ "system_prompt = \"\"\"\n",
+ "For a given user input, rewrite the input to make be more concise.\n",
+ "\"\"\"\n",
+ "\n",
+ "# user input for re-writing questions\n",
+ "questions = [\n",
+ " \"LiteLLM is a lightweight Python package that simplifies the process of making API calls to various language models. Here are some reasons why you should use LiteLLM:\\n\\n1. **Simplified API Calls**: LiteLLM abstracts away the complexity of making API calls to different language models. It provides a unified interface for invoking models from OpenAI, Azure, Cohere, Anthropic, Huggingface, and more.\\n\\n2. **Easy Integration**: LiteLLM seamlessly integrates with your existing codebase. You can import the package and start making API calls with just a few lines of code.\\n\\n3. **Flexibility**: LiteLLM supports a variety of language models, including GPT-3, GPT-Neo, chatGPT, and more. You can choose the model that suits your requirements and easily switch between them.\\n\\n4. **Convenience**: LiteLLM handles the authentication and connection details for you. You just need to set the relevant environment variables, and the package takes care of the rest.\\n\\n5. **Quick Prototyping**: LiteLLM is ideal for rapid prototyping and experimentation. With its simple API, you can quickly generate text, chat with models, and build interactive applications.\\n\\n6. **Community Support**: LiteLLM is actively maintained and supported by a community of developers. You can find help, share ideas, and collaborate with others to enhance your projects.\\n\\nOverall, LiteLLM simplifies the process of making API calls to language models, saving you time and effort while providing flexibility and convenience\",\n",
+ " \"Hi everyone! I'm [your name] and I'm currently working on [your project/role involving LLMs]. I came across LiteLLM and was really excited by how it simplifies working with different LLM providers. I'm hoping to use LiteLLM to [build an app/simplify my code/test different models etc]. Before finding LiteLLM, I was struggling with [describe any issues you faced working with multiple LLMs]. With LiteLLM's unified API and automatic translation between providers, I think it will really help me to [goals you have for using LiteLLM]. Looking forward to being part of this community and learning more about how I can build impactful applications powered by LLMs!Let me know if you would like me to modify or expand on any part of this suggested intro. I'm happy to provide any clarification or additional details you need!\",\n",
+ " \"Traceloop is a platform for monitoring and debugging the quality of your LLM outputs. It provides you with a way to track the performance of your LLM application; rollout changes with confidence; and debug issues in production. It is based on OpenTelemetry, so it can provide full visibility to your LLM requests, as well vector DB usage, and other infra in your stack.\"\n",
+ "]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "fwNcC_obICUc"
+ },
+ "source": [
+ "## Run Questions"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "KtBjZ1mUIBiJ"
+ },
+ "outputs": [],
+ "source": [
+ "from litellm import completion, completion_cost\n",
+ "import os\n",
+ "import time\n",
+ "\n",
+ "# optional use litellm dashboard to view logs\n",
+ "# litellm.use_client = True\n",
+ "# litellm.token = \"ishaan_2@berri.ai\" # set your email\n",
+ "\n",
+ "os.environ['TOGETHERAI_API_KEY'] = \"\"\n",
+ "os.environ['OPENAI_API_KEY'] = \"\"\n",
+ "os.environ['ANTHROPIC_API_KEY'] = \"\"\n",
+ "\n",
+ "models = ['togethercomputer/llama-2-70b-chat', 'gpt-3.5-turbo', 'claude-instant-1.2'] # enter llms to benchmark\n",
+ "data_2 = []\n",
+ "\n",
+ "for question in questions: # group by question\n",
+ " for model in models:\n",
+ " print(f\"running question: {question} for model: {model}\")\n",
+ " start_time = time.time()\n",
+ " # show response, response time, cost for each question\n",
+ " response = completion(\n",
+ " model=model,\n",
+ " max_tokens=500,\n",
+ " messages = [\n",
+ " {\n",
+ " \"role\": \"system\", \"content\": system_prompt\n",
+ " },\n",
+ " {\n",
+ " \"role\": \"user\", \"content\": \"User input:\" + question\n",
+ " }\n",
+ " ],\n",
+ " )\n",
+ " end = time.time()\n",
+ " total_time = end-start_time # response time\n",
+ " # print(response)\n",
+ " cost = completion_cost(response) # cost for completion\n",
+ " raw_response = response['choices'][0]['message']['content'] # response string\n",
+ " #print(raw_response, total_time, cost)\n",
+ "\n",
+ " # add to pandas df\n",
+ " data_2.append(\n",
+ " {\n",
+ " 'Model': model,\n",
+ " 'Question': question,\n",
+ " 'Response': raw_response,\n",
+ " 'ResponseTime': total_time,\n",
+ " 'Cost': cost\n",
+ " })\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "-PCYIzG5M0II"
+ },
+ "source": [
+ "## View Logs - Group by Question"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1000
+ },
+ "id": "-3R5-2q8IiL2",
+ "outputId": "c4a0d9e5-bb21-4de0-fc4c-9f5e71d0f177"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Question: Hi everyone! I'm [your name] and I'm currently working on [your project/role involving LLMs]. I came across LiteLLM and was really excited by how it simplifies working with different LLM providers. I'm hoping to use LiteLLM to [build an app/simplify my code/test different models etc]. Before finding LiteLLM, I was struggling with [describe any issues you faced working with multiple LLMs]. With LiteLLM's unified API and automatic translation between providers, I think it will really help me to [goals you have for using LiteLLM]. Looking forward to being part of this community and learning more about how I can build impactful applications powered by LLMs!Let me know if you would like me to modify or expand on any part of this suggested intro. I'm happy to provide any clarification or additional details you need!\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "
Model
\n",
+ "
Question
\n",
+ "
Response
\n",
+ "
ResponseTime
\n",
+ "
Cost
\n",
+ "
\n",
+ " \n",
+ " \n",
+ "
\n",
+ "
3
\n",
+ "
togethercomputer/llama-2-70b-chat
\n",
+ "
Hi everyone! I'm [your name] and I'm currently working on [your project/role involving LLMs]. I came across LiteLLM and was really excited by how it simplifies working with different LLM providers. I'm hoping to use LiteLLM to [build an app/simplify my code/test different models etc]. Before finding LiteLLM, I was struggling with [describe any issues you faced working with multiple LLMs]. With LiteLLM's unified API and automatic translation between providers, I think it will really help me to [goals you have for using LiteLLM]. Looking forward to being part of this community and learning more about how I can build impactful applications powered by LLMs!Let me know if you would like me to modify or expand on any part of this suggested intro. I'm happy to provide any clarification or additional details you need!
\n",
+ "
\\nHere's a more concise version of the user input:\\n\\n\"Hi everyone! I'm [your name] and I'm working on [your project/role involving LLMs]. I recently discovered LiteLLM and I'm excited to use it to [build an app/simplify my code/test different models etc]. Before LiteLLM, I struggled with [describe any issues you faced working with multiple LLMs]. I'm looking forward to using LiteLLM's unified API and automatic translation to achieve my goals. I'm eager to learn more about building impactful applications powered by LLMs and to be part of this community. Let me know if you have any questions or need further clarification.\"\\n\\nIn this revised version, we've kept the essential information and removed some of the extraneous language. We've also rephrased some of the sentences to make them more concise and easier to read.
\n",
+ "
18.300620
\n",
+ "
0.001200
\n",
+ "
\n",
+ "
\n",
+ "
4
\n",
+ "
gpt-3.5-turbo
\n",
+ "
Hi everyone! I'm [your name] and I'm currently working on [your project/role involving LLMs]. I came across LiteLLM and was really excited by how it simplifies working with different LLM providers. I'm hoping to use LiteLLM to [build an app/simplify my code/test different models etc]. Before finding LiteLLM, I was struggling with [describe any issues you faced working with multiple LLMs]. With LiteLLM's unified API and automatic translation between providers, I think it will really help me to [goals you have for using LiteLLM]. Looking forward to being part of this community and learning more about how I can build impactful applications powered by LLMs!Let me know if you would like me to modify or expand on any part of this suggested intro. I'm happy to provide any clarification or additional details you need!
\n",
+ "
User input: Hi, I'm [your name] and I'm excited about using LiteLLM to simplify working with different LLM providers. Before finding LiteLLM, I faced challenges working with multiple LLMs. With LiteLLM's unified API and automatic translation, I believe it will help me achieve my goals of [state your goals]. I look forward to being part of this community and learning how to build impactful applications with LLMs. Let me know if you need any further clarification or details.
\n",
+ "
7.385472
\n",
+ "
0.000525
\n",
+ "
\n",
+ "
\n",
+ "
5
\n",
+ "
claude-instant-1.2
\n",
+ "
Hi everyone! I'm [your name] and I'm currently working on [your project/role involving LLMs]. I came across LiteLLM and was really excited by how it simplifies working with different LLM providers. I'm hoping to use LiteLLM to [build an app/simplify my code/test different models etc]. Before finding LiteLLM, I was struggling with [describe any issues you faced working with multiple LLMs]. With LiteLLM's unified API and automatic translation between providers, I think it will really help me to [goals you have for using LiteLLM]. Looking forward to being part of this community and learning more about how I can build impactful applications powered by LLMs!Let me know if you would like me to modify or expand on any part of this suggested intro. I'm happy to provide any clarification or additional details you need!
\n",
+ "
Here is a more concise rewrite of the user input:\\n\\nHi everyone, I'm [your name]. I'm currently [your project/role] and came across LiteLLM, which simplifies working with different LLMs through its unified API. I hope to [build an app/simplify code/test models] with LiteLLM since I previously struggled with [issues]. LiteLLM's automatic translation between providers will help me [goals] and build impactful LLM applications. Looking forward to learning more as part of this community. Let me know if you need any clarification on my plans to use LiteLLM.
\n",
+ "
8.628217
\n",
+ "
0.001022
\n",
+ "
\n",
+ " \n",
+ "
"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "execution_count": 20,
+ "metadata": {},
+ "output_type": "execute_result"
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Question: LiteLLM is a lightweight Python package that simplifies the process of making API calls to various language models. Here are some reasons why you should use LiteLLM:\n",
+ "\n",
+ "1. **Simplified API Calls**: LiteLLM abstracts away the complexity of making API calls to different language models. It provides a unified interface for invoking models from OpenAI, Azure, Cohere, Anthropic, Huggingface, and more.\n",
+ "\n",
+ "2. **Easy Integration**: LiteLLM seamlessly integrates with your existing codebase. You can import the package and start making API calls with just a few lines of code.\n",
+ "\n",
+ "3. **Flexibility**: LiteLLM supports a variety of language models, including GPT-3, GPT-Neo, chatGPT, and more. You can choose the model that suits your requirements and easily switch between them.\n",
+ "\n",
+ "4. **Convenience**: LiteLLM handles the authentication and connection details for you. You just need to set the relevant environment variables, and the package takes care of the rest.\n",
+ "\n",
+ "5. **Quick Prototyping**: LiteLLM is ideal for rapid prototyping and experimentation. With its simple API, you can quickly generate text, chat with models, and build interactive applications.\n",
+ "\n",
+ "6. **Community Support**: LiteLLM is actively maintained and supported by a community of developers. You can find help, share ideas, and collaborate with others to enhance your projects.\n",
+ "\n",
+ "Overall, LiteLLM simplifies the process of making API calls to language models, saving you time and effort while providing flexibility and convenience\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "
Model
\n",
+ "
Question
\n",
+ "
Response
\n",
+ "
ResponseTime
\n",
+ "
Cost
\n",
+ "
\n",
+ " \n",
+ " \n",
+ "
\n",
+ "
0
\n",
+ "
togethercomputer/llama-2-70b-chat
\n",
+ "
LiteLLM is a lightweight Python package that simplifies the process of making API calls to various language models. Here are some reasons why you should use LiteLLM:\\n\\n1. **Simplified API Calls**: LiteLLM abstracts away the complexity of making API calls to different language models. It provides a unified interface for invoking models from OpenAI, Azure, Cohere, Anthropic, Huggingface, and more.\\n\\n2. **Easy Integration**: LiteLLM seamlessly integrates with your existing codebase. You can import the package and start making API calls with just a few lines of code.\\n\\n3. **Flexibility**: LiteLLM supports a variety of language models, including GPT-3, GPT-Neo, chatGPT, and more. You can choose the model that suits your requirements and easily switch between them.\\n\\n4. **Convenience**: LiteLLM handles the authentication and connection details for you. You just need to set the relevant environment variables, and the package takes care of the rest.\\n\\n5. **Quick Prototyping**: LiteLLM is ideal for rapid prototyping and experimentation. With its simple API, you can quickly generate text, chat with models, and build interactive applications.\\n\\n6. **Community Support**: LiteLLM is actively maintained and supported by a community of developers. You can find help, share ideas, and collaborate with others to enhance your projects.\\n\\nOverall, LiteLLM simplifies the process of making API calls to language models, saving you time and effort while providing flexibility and convenience
\n",
+ "
Here's a more concise version of the user input:\\n\\nLiteLLM is a lightweight Python package that simplifies API calls to various language models. It abstracts away complexity, integrates seamlessly, supports multiple models, and handles authentication. It's ideal for rapid prototyping and has community support. It saves time and effort while providing flexibility and convenience.
\n",
+ "
11.294250
\n",
+ "
0.001251
\n",
+ "
\n",
+ "
\n",
+ "
1
\n",
+ "
gpt-3.5-turbo
\n",
+ "
LiteLLM is a lightweight Python package that simplifies the process of making API calls to various language models. Here are some reasons why you should use LiteLLM:\\n\\n1. **Simplified API Calls**: LiteLLM abstracts away the complexity of making API calls to different language models. It provides a unified interface for invoking models from OpenAI, Azure, Cohere, Anthropic, Huggingface, and more.\\n\\n2. **Easy Integration**: LiteLLM seamlessly integrates with your existing codebase. You can import the package and start making API calls with just a few lines of code.\\n\\n3. **Flexibility**: LiteLLM supports a variety of language models, including GPT-3, GPT-Neo, chatGPT, and more. You can choose the model that suits your requirements and easily switch between them.\\n\\n4. **Convenience**: LiteLLM handles the authentication and connection details for you. You just need to set the relevant environment variables, and the package takes care of the rest.\\n\\n5. **Quick Prototyping**: LiteLLM is ideal for rapid prototyping and experimentation. With its simple API, you can quickly generate text, chat with models, and build interactive applications.\\n\\n6. **Community Support**: LiteLLM is actively maintained and supported by a community of developers. You can find help, share ideas, and collaborate with others to enhance your projects.\\n\\nOverall, LiteLLM simplifies the process of making API calls to language models, saving you time and effort while providing flexibility and convenience
\n",
+ "
LiteLLM is a lightweight Python package that simplifies API calls to various language models. Here's why you should use it:\\n1. Simplified API Calls: Works with multiple models (OpenAI, Azure, Cohere, Anthropic, Huggingface).\\n2. Easy Integration: Import and start using it quickly in your codebase.\\n3. Flexibility: Supports GPT-3, GPT-Neo, chatGPT, etc. easily switch between models.\\n4. Convenience: Handles authentication and connection details, just set environment variables.\\n5. Quick Prototyping: Great for rapid prototyping and building interactive applications.\\n6. Community Support: Actively maintained and supported by a developer community.
\n",
+ "
9.778315
\n",
+ "
0.000795
\n",
+ "
\n",
+ "
\n",
+ "
2
\n",
+ "
claude-instant-1.2
\n",
+ "
LiteLLM is a lightweight Python package that simplifies the process of making API calls to various language models. Here are some reasons why you should use LiteLLM:\\n\\n1. **Simplified API Calls**: LiteLLM abstracts away the complexity of making API calls to different language models. It provides a unified interface for invoking models from OpenAI, Azure, Cohere, Anthropic, Huggingface, and more.\\n\\n2. **Easy Integration**: LiteLLM seamlessly integrates with your existing codebase. You can import the package and start making API calls with just a few lines of code.\\n\\n3. **Flexibility**: LiteLLM supports a variety of language models, including GPT-3, GPT-Neo, chatGPT, and more. You can choose the model that suits your requirements and easily switch between them.\\n\\n4. **Convenience**: LiteLLM handles the authentication and connection details for you. You just need to set the relevant environment variables, and the package takes care of the rest.\\n\\n5. **Quick Prototyping**: LiteLLM is ideal for rapid prototyping and experimentation. With its simple API, you can quickly generate text, chat with models, and build interactive applications.\\n\\n6. **Community Support**: LiteLLM is actively maintained and supported by a community of developers. You can find help, share ideas, and collaborate with others to enhance your projects.\\n\\nOverall, LiteLLM simplifies the process of making API calls to language models, saving you time and effort while providing flexibility and convenience
\n",
+ "
Here is a concise rewrite of the user input:\\n\\nLiteLLM is a lightweight Python package that simplifies accessing various language models. It provides a unified interface for models from OpenAI, Azure, Cohere, Anthropic, Huggingface, and more. Key benefits include simplified API calls, easy integration, flexibility to use different models, automated handling of authentication, and support for quick prototyping. The actively maintained package saves time by abstracting away complexity while offering convenience and a collaborative community.
\n",
+ "
7.697520
\n",
+ "
0.001098
\n",
+ "
\n",
+ " \n",
+ "
"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "execution_count": 20,
+ "metadata": {},
+ "output_type": "execute_result"
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Question: Traceloop is a platform for monitoring and debugging the quality of your LLM outputs. It provides you with a way to track the performance of your LLM application; rollout changes with confidence; and debug issues in production. It is based on OpenTelemetry, so it can provide full visibility to your LLM requests, as well vector DB usage, and other infra in your stack.\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "
Model
\n",
+ "
Question
\n",
+ "
Response
\n",
+ "
ResponseTime
\n",
+ "
Cost
\n",
+ "
\n",
+ " \n",
+ " \n",
+ "
\n",
+ "
6
\n",
+ "
togethercomputer/llama-2-70b-chat
\n",
+ "
Traceloop is a platform for monitoring and debugging the quality of your LLM outputs. It provides you with a way to track the performance of your LLM application; rollout changes with confidence; and debug issues in production. It is based on OpenTelemetry, so it can provide full visibility to your LLM requests, as well vector DB usage, and other infra in your stack.
\n",
+ "
\\nRewritten input: Traceloop is a platform for monitoring and debugging LLM outputs. It allows users to track performance, rollout changes confidently, and debug issues in production. It uses OpenTelemetry for full visibility into LLM requests, vector DB usage, and other infrastructure.
\n",
+ "
9.060444
\n",
+ "
0.000525
\n",
+ "
\n",
+ "
\n",
+ "
7
\n",
+ "
gpt-3.5-turbo
\n",
+ "
Traceloop is a platform for monitoring and debugging the quality of your LLM outputs. It provides you with a way to track the performance of your LLM application; rollout changes with confidence; and debug issues in production. It is based on OpenTelemetry, so it can provide full visibility to your LLM requests, as well vector DB usage, and other infra in your stack.
\n",
+ "
Traceloop is a platform for monitoring and debugging the quality of your LLM outputs. It helps track performance, rollout changes, and debug issues in production. It is based on OpenTelemetry, providing visibility to LLM requests, vector DB usage, and other infrastructure in your stack.
\n",
+ "
7.304661
\n",
+ "
0.000283
\n",
+ "
\n",
+ "
\n",
+ "
8
\n",
+ "
claude-instant-1.2
\n",
+ "
Traceloop is a platform for monitoring and debugging the quality of your LLM outputs. It provides you with a way to track the performance of your LLM application; rollout changes with confidence; and debug issues in production. It is based on OpenTelemetry, so it can provide full visibility to your LLM requests, as well vector DB usage, and other infra in your stack.
\n",
+ "
Here is a more concise rewrite of the user input:\\n\\nTraceloop monitors and debugs LLM quality. It tracks LLM performance, enables confident changes, and debugs production issues. Based on OpenTelemetry, Traceloop provides full visibility into LLM requests, vector DB usage, and other stack infrastructure.
\n"
+ ],
+ "text/plain": [
+ "Model Name claude-instant-1 \\\n",
+ "Prompt \n",
+ "\\nIs paul graham a writer? Yes, Paul Graham is considered a writer in ad... \n",
+ "\\nWhat has Paul Graham done? Paul Graham has made significant contribution... \n",
+ "\\nWhat is Paul Graham known for? Paul Graham is known for several things:\\n\\n-... \n",
+ "\\nWhere does Paul Graham live? Based on the information provided:\\n\\n- Paul ... \n",
+ "\\nWho is Paul Graham? Paul Graham is an influential computer scient... \n",
+ "\n",
+ "Model Name gpt-3.5-turbo-0613 \\\n",
+ "Prompt \n",
+ "\\nIs paul graham a writer? Yes, Paul Graham is a writer. He has written s... \n",
+ "\\nWhat has Paul Graham done? Paul Graham has achieved several notable accom... \n",
+ "\\nWhat is Paul Graham known for? Paul Graham is known for his work on the progr... \n",
+ "\\nWhere does Paul Graham live? According to the given information, Paul Graha... \n",
+ "\\nWho is Paul Graham? Paul Graham is an English computer scientist, ... \n",
+ "\n",
+ "Model Name gpt-3.5-turbo-16k-0613 \\\n",
+ "Prompt \n",
+ "\\nIs paul graham a writer? Yes, Paul Graham is a writer. He has authored ... \n",
+ "\\nWhat has Paul Graham done? Paul Graham has made significant contributions... \n",
+ "\\nWhat is Paul Graham known for? Paul Graham is known for his work on the progr... \n",
+ "\\nWhere does Paul Graham live? Paul Graham currently lives in England, where ... \n",
+ "\\nWho is Paul Graham? Paul Graham is an English computer scientist, ... \n",
+ "\n",
+ "Model Name gpt-4-0613 \\\n",
+ "Prompt \n",
+ "\\nIs paul graham a writer? Yes, Paul Graham is a writer. He is an essayis... \n",
+ "\\nWhat has Paul Graham done? Paul Graham is known for his work on the progr... \n",
+ "\\nWhat is Paul Graham known for? Paul Graham is known for his work on the progr... \n",
+ "\\nWhere does Paul Graham live? The text does not provide a current place of r... \n",
+ "\\nWho is Paul Graham? Paul Graham is an English computer scientist, ... \n",
+ "\n",
+ "Model Name replicate/llama-2-70b-chat:58d078176e02c219e11eb4da5a02a7830a283b14cf8f94537af893ccff5ee781 \n",
+ "Prompt \n",
+ "\\nIs paul graham a writer? Yes, Paul Graham is an author. According to t... \n",
+ "\\nWhat has Paul Graham done? Paul Graham has had a diverse career in compu... \n",
+ "\\nWhat is Paul Graham known for? Paul Graham is known for many things, includi... \n",
+ "\\nWhere does Paul Graham live? Based on the information provided, Paul Graha... \n",
+ "\\nWho is Paul Graham? Paul Graham is an English computer scientist,... "
+ ]
+ },
+ "execution_count": 17,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import pandas as pd\n",
+ "\n",
+ "# Create an empty list to store the row data\n",
+ "table_data = []\n",
+ "\n",
+ "# Iterate through the list and extract the required data\n",
+ "for item in result:\n",
+ " prompt = item['prompt'][0]['content'].replace(context, \"\") # clean the prompt for easy comparison\n",
+ " model = item['response']['model']\n",
+ " response = item['response']['choices'][0]['message']['content']\n",
+ " table_data.append([prompt, model, response])\n",
+ "\n",
+ "# Create a DataFrame from the table data\n",
+ "df = pd.DataFrame(table_data, columns=['Prompt', 'Model Name', 'Response'])\n",
+ "\n",
+ "# Pivot the DataFrame to get the desired table format\n",
+ "table = df.pivot(index='Prompt', columns='Model Name', values='Response')\n",
+ "table"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "zOxUM40PINDC"
+ },
+ "source": [
+ "# Load Test endpoint\n",
+ "\n",
+ "Run 100+ simultaneous queries across multiple providers to see when they fail + impact on latency"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ZkQf_wbcIRQ9"
+ },
+ "outputs": [],
+ "source": [
+ "models=[\"gpt-3.5-turbo\", \"replicate/llama-2-70b-chat:58d078176e02c219e11eb4da5a02a7830a283b14cf8f94537af893ccff5ee781\", \"claude-instant-1\"]\n",
+ "context = \"\"\"Paul Graham (/ɔrƦm/; born 1964)[3] is an English computer scientist, essayist, entrepreneur, venture capitalist, and author. He is best known for his work on the programming language Lisp, his former startup Viaweb (later renamed Yahoo! Store), cofounding the influential startup accelerator and seed capital firm Y Combinator, his essays, and Hacker News. He is the author of several computer programming books, including: On Lisp,[4] ANSI Common Lisp,[5] and Hackers & Painters.[6] Technology journalist Steven Levy has described Graham as a \"hacker philosopher\".[7] Graham was born in England, where he and his family maintain permanent residence. However he is also a citizen of the United States, where he was educated, lived, and worked until 2016.\"\"\"\n",
+ "prompt = \"Where does Paul Graham live?\"\n",
+ "final_prompt = context + prompt\n",
+ "result = load_test_model(models=models, prompt=final_prompt, num_calls=5)"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8vSNBFC06aXY"
+ },
+ "source": [
+ "## Visualize the data"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 552
+ },
+ "id": "SZfiKjLV3-n8",
+ "outputId": "00f7f589-b3da-43ed-e982-f9420f074b8d"
+ },
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAIXCAYAAACy1HXAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABn5UlEQVR4nO3dd1QT2d8G8Cf0ojQBEUFRsSv2FXvvvSx2saNi7733ihXELotd7KuIir33sjZUsIuKVGmS+/7hy/yM6K7RYEZ4PufkaO5Mkm/IJHly594ZhRBCgIiIiEiGdLRdABEREdG3MKgQERGRbDGoEBERkWwxqBAREZFsMagQERGRbDGoEBERkWwxqBAREZFsMagQERGRbDGoEBERkWwxqBCR7Dk5OaFLly7aLkNtc+fORd68eaGrq4uSJUtquxyNO3bsGBQKBbZv367tUtSmUCgwadIktW8XGhoKhUKBdevWabwm+joGFVKxfPlyKBQKlC9fXtulyI6TkxMUCoV0MTU1xR9//IENGzZou7TfTuoX3PdcfleHDh3CiBEjUKlSJaxduxYzZszQdkmys27dOul1PnXqVJrlQgg4OjpCoVCgcePGWqiQ5EBP2wWQvPj7+8PJyQkXLlxASEgInJ2dtV2SrJQsWRJDhw4FALx8+RKrVq2Cu7s7EhMT0bNnTy1X9/soXLgw/Pz8VNpGjx6NLFmyYOzYsWnWv3fvHnR0fq/fVUePHoWOjg5Wr14NAwMDbZcja0ZGRti4cSMqV66s0n78+HE8e/YMhoaGWqqM5IBBhSSPHz/GmTNnEBAQAA8PD/j7+2PixIm/tAalUomkpCQYGRn90sf9Xjlz5kTHjh2l6126dEHevHmxcOFCBhU1ZM+eXeXvCACzZs2CtbV1mnYAv+UXVXh4OIyNjTUWUoQQSEhIgLGxsUbuT04aNmyIbdu2YfHixdDT+9/X0saNG1GmTBm8fftWi9WRtv1eP1EoXfn7+8PS0hKNGjVC69at4e/vLy1LTk6GlZUVunbtmuZ20dHRMDIywrBhw6S2xMRETJw4Ec7OzjA0NISjoyNGjBiBxMREldsqFAr069cP/v7+KFq0KAwNDXHw4EEAwLx581CxYkVky5YNxsbGKFOmzFf3hcfHx2PAgAGwtrZG1qxZ0bRpUzx//vyr+6CfP3+Obt26IXv27DA0NETRokWxZs2aH/6b2djYoFChQnj48KFKu1KphJeXF4oWLQojIyNkz54dHh4eeP/+vcp6ly5dQr169WBtbQ1jY2PkyZMH3bp1k5an7g+fN28eFi5ciNy5c8PY2BjVqlXDrVu30tRz9OhRVKlSBaamprCwsECzZs1w584dlXUmTZoEhUKBkJAQdOnSBRYWFjA3N0fXrl3x4cMHlXWDgoJQuXJlWFhYIEuWLChYsCDGjBmjss73vtY/48sxKqm7DE6dOoUBAwbAxsYGFhYW8PDwQFJSEiIjI9G5c2dYWlrC0tISI0aMwJcnitfUa/Q1CoUCa9euRVxcnLRrI3VMw8ePHzF16lTky5cPhoaGcHJywpgxY9L8vZycnNC4cWMEBgaibNmyMDY2xooVK/71cc+fP4/69evD3NwcJiYmqFatGk6fPq2yTlhYGPr27YuCBQvC2NgY2bJlw59//onQ0NA09xcZGYnBgwfDyckJhoaGcHBwQOfOndMEB6VSienTp8PBwQFGRkaoVasWQkJC/rXWz7Vr1w7v3r1DUFCQ1JaUlITt27ejffv2X71NXFwchg4dCkdHRxgaGqJgwYKYN29emtc5MTERgwcPho2NjfT58OzZs6/ep6Y/H0hDBNH/K1SokOjevbsQQogTJ04IAOLChQvS8m7dugkLCwuRmJiocrv169cLAOLixYtCCCFSUlJE3bp1hYmJiRg0aJBYsWKF6Nevn9DT0xPNmjVTuS0AUbhwYWFjYyMmT54sli1bJq5evSqEEMLBwUH07dtXLF26VCxYsED88ccfAoDYt2+fyn24ubkJAKJTp05i2bJlws3NTZQoUUIAEBMnTpTWe/XqlXBwcBCOjo5iypQpwtvbWzRt2lQAEAsXLvzPv0/u3LlFo0aNVNqSk5OFnZ2dyJ49u0p7jx49hJ6enujZs6fw8fERI0eOFKampqJcuXIiKSlJCCHE69evhaWlpShQoICYO3euWLlypRg7dqwoXLiwdD+PHz8WAETx4sWFk5OTmD17tpg8ebKwsrISNjY24tWrV9K6QUFBQk9PTxQoUEDMmTNHTJ48WVhbWwtLS0vx+PFjab2JEycKAKJUqVKiZcuWYvny5aJHjx4CgBgxYoS03q1bt4SBgYEoW7asWLRokfDx8RHDhg0TVatWldZR57X+L0WLFhXVqlX75t/e3d1dur527VoBQJQsWVLUr19fLFu2THTq1El6DpUrVxbt27cXy5cvF40bNxYAxPr169PlNfoaPz8/UaVKFWFoaCj8/PyEn5+fePjwoRBCCHd3dwFAtG7dWixbtkx07txZABDNmzdP85ydnZ2FpaWlGDVqlPDx8RHBwcHffMwjR44IAwMDUaFCBTF//nyxcOFC4eLiIgwMDMT58+el9bZt2yZKlCghJkyYIHx9fcWYMWOEpaWlyJ07t4iLi5PWi4mJEcWKFRO6urqiZ8+ewtvbW0ydOlWUK1dOeo8GBwdL21KZMmXEwoULxaRJk4SJiYn4448//vVv9PnrePHiRVGxYkXRqVMnadmuXbuEjo6OeP78eZr3nlKpFDVr1hQKhUL06NFDLF26VDRp0kQAEIMGDVJ5jI4dOwoAon379mLp0qWiZcuWwsXF5Yc/H1Lfk2vXrv3P50eawaBCQgghLl26JACIoKAgIcSnDwIHBwcxcOBAaZ3AwEABQOzdu1fltg0bNhR58+aVrvv5+QkdHR1x8uRJlfV8fHwEAHH69GmpDYDQ0dERt2/fTlPThw8fVK4nJSWJYsWKiZo1a0ptly9f/uqHU5cuXdJ8EHXv3l3kyJFDvH37VmXdtm3bCnNz8zSP96XcuXOLunXrijdv3og3b96ImzdvSl+Onp6e0nonT54UAIS/v7/K7Q8ePKjSvnPnTpWA9zWpH4rGxsbi2bNnUvv58+cFADF48GCprWTJksLW1la8e/dOart+/brQ0dERnTt3ltpSg0q3bt1UHqtFixYiW7Zs0vWFCxcKAOLNmzffrE+d1/q//EhQqVevnlAqlVJ7hQoVhEKhEL1795baPn78KBwcHFTuW5Ov0be4u7sLU1NTlbZr164JAKJHjx4q7cOGDRMAxNGjR1WeMwBx8ODB/3wspVIp8ufPn+bv8eHDB5EnTx5Rp04dlbYvnT17VgAQGzZskNomTJggAIiAgICvPp4Q/wsqhQsXVvkBs2jRIgFA3Lx581/r/jyoLF26VGTNmlWq788//xQ1atSQ/hafB5Vdu3YJAGLatGkq99e6dWuhUChESEiIEOJ/f+++ffuqrNe+ffsf/nxgUPn1uOuHAHza7ZM9e3bUqFEDwKeu6zZt2mDz5s1ISUkBANSsWRPW1tbYsmWLdLv3798jKCgIbdq0kdq2bduGwoULo1ChQnj79q10qVmzJgAgODhY5bGrVauGIkWKpKnp833x79+/R1RUFKpUqYIrV65I7am7ifr27aty2/79+6tcF0Jgx44daNKkCYQQKnXVq1cPUVFRKvf7LYcOHYKNjQ1sbGxQvHhx+Pn5oWvXrpg7d67K8zc3N0edOnVUHqdMmTLIkiWL9PwtLCwAAPv27UNycvK/Pm7z5s2RM2dO6foff/yB8uXL4++//wbwaWDvtWvX0KVLF1hZWUnrubi4oE6dOtJ6n+vdu7fK9SpVquDdu3eIjo5WqW/37t1QKpVfrUvd11rTunfvrjIzqHz58hBCoHv37lKbrq4uypYti0ePHqnUrenX6Hukvg5DhgxRaU8doL1//36V9jx58qBevXr/eb/Xrl3DgwcP0L59e7x79056PnFxcahVqxZOnDghvYafv6+Sk5Px7t07ODs7w8LCQuU9sGPHDpQoUQItWrRI83hfzsbq2rWrylicKlWqAIDK3/y/uLm5IT4+Hvv27UNMTAz27dv3zd0+f//9N3R1dTFgwACV9qFDh0IIgQMHDkjrAUiz3qBBg1Sua+rzgdJHhgkqJ06cQJMmTWBvbw+FQoFdu3al+2M+f/4cHTt2lMZQFC9eHJcuXUr3x9W0lJQUbN68GTVq1MDjx48REhKCkJAQlC9fHq9fv8aRI0cAAHp6emjVqhV2794t7U8PCAhAcnKySlB58OABbt++LX2hp14KFCgA4NMgw8/lyZPnq3Xt27cPrq6uMDIygpWVFWxsbODt7Y2oqChpnbCwMOjo6KS5jy9nK7158waRkZHw9fVNU1fquJsv6/qa8uXLIygoCAcPHsS8efNgYWGB9+/fq3xIP3jwAFFRUbC1tU3zWLGxsdLjVKtWDa1atcLkyZNhbW2NZs2aYe3atV8d25E/f/40bQUKFJDGFYSFhQEAChYsmGa9woULS19an8uVK5fKdUtLSwCQxmi0adMGlSpVQo8ePZA9e3a0bdsWW7duVQkt6r7WmvblczA3NwcAODo6pmn/fOxJerxG3yN1e/1y+7Szs4OFhYX0Oqb61nvjSw8ePAAAuLu7p3k+q1atQmJiovS+iY+Px4QJE6SxHdbW1rCxsUFkZKTKe+vhw4coVqzYdz3+f21L38PGxga1a9fGxo0bERAQgJSUFLRu3fqr64aFhcHe3h5Zs2ZVaS9cuLC0PPVfHR0d5MuXT2W9L98nmvp8oPSRYWb9xMXFoUSJEujWrRtatmyZ7o/3/v17VKpUCTVq1MCBAwdgY2ODBw8eSG/Q38nRo0fx8uVLbN68GZs3b06z3N/fH3Xr1gUAtG3bFitWrMCBAwfQvHlzbN26FYUKFUKJEiWk9ZVKJYoXL44FCxZ89fG+/BL52iyGkydPomnTpqhatSqWL1+OHDlyQF9fH2vXrsXGjRvVfo6pX64dO3aEu7v7V9dxcXH5z/uxtrZG7dq1AQD16tVDoUKF0LhxYyxatEj6laxUKmFra6syGPlzNjY2ACAdKOvcuXPYu3cvAgMD0a1bN8yfPx/nzp1DlixZ1H6e6tDV1f1qu/j/wYjGxsY4ceIEgoODsX//fhw8eBBbtmxBzZo1cejQIejq6qr9Wmvat57D19rFZ4Mstf0afe/xYb53hk/q9j137txvHlgutdb+/ftj7dq1GDRoECpUqABzc3MoFAq0bdv2mz1n/+W/tqXv1b59e/Ts2ROvXr1CgwYNpB6t9KapzwdKHxkmqDRo0AANGjT45vLExESMHTsWmzZtQmRkJIoVK4bZs2ejevXqP/R4s2fPhqOjI9auXSu1fe+vH7nx9/eHra0tli1blmZZQEAAdu7cCR8fHxgbG6Nq1arIkSMHtmzZgsqVK+Po0aNpjnuRL18+XL9+HbVq1frhA3bt2LEDRkZGCAwMVJma+vnfGwBy584NpVKJx48fq/Q6fDnjIHXEf0pKihQ0NKFRo0aoVq0aZsyYAQ8PD5iamiJfvnw4fPgwKlWq9F1fNK6urnB1dcX06dOxceNGdOjQAZs3b0aPHj2kdVJ/MX/u/v37cHJyAvDp7wB8Ot7Il+7evQtra2uYmpqq/fx0dHRQq1Yt1KpVCwsWLMCMGTMwduxYBAcHo3bt2hp5rbUhPV6j75G6vT548ED69Q8Ar1+/RmRkpPQ6qiu1x8DMzOw/t+/t27fD3d0d8+fPl9oSEhIQGRmZ5j6/NrMsPbVo0QIeHh44d+6cyi7mL+XOnRuHDx9GTEyMSq/K3bt3peWp/yqVSjx8+FClF+XL90l6fT6QZmSYXT//pV+/fjh79iw2b96MGzdu4M8//0T9+vW/+gXwPfbs2YOyZcvizz//hK2tLUqVKoWVK1dquOr0Fx8fj4CAADRu3BitW7dOc+nXrx9iYmKwZ88eAJ++uFq3bo29e/fCz88PHz9+VNntA3za1/z8+fOv/j3i4+PT7IL4Gl1dXSgUCml8DPBpqu6Xu/RS998vX75cpX3JkiVp7q9Vq1bYsWPHVz9837x58581fcvIkSPx7t076fm6ubkhJSUFU6dOTbPux48fpS+E9+/fp/nFmfpr+MtdC7t27cLz58+l6xcuXMD58+elcJ4jRw6ULFkS69evV/nCuXXrFg4dOoSGDRuq/bwiIiLStH1ZnyZea21Ij9foe6S+Dl5eXirtqT1SjRo1Uvs+AaBMmTLIly8f5s2bh9jY2DTLP9++dXV10zynJUuWqLzXAKBVq1a4fv06du7cmeb+1O0p+V5ZsmSBt7c3Jk2ahCZNmnxzvYYNGyIlJQVLly5VaV+4cCEUCoX0vkj9d/HixSrrffn3T8/PB/p5GaZH5d88efIEa9euxZMnT2Bvbw8AGDZsGA4ePPjDh7Z+9OgRvL29MWTIEIwZMwYXL17EgAEDYGBg8M2uQznas2cPYmJi0LRp068ud3V1hY2NDfz9/aVA0qZNGyxZsgQTJ05E8eLFVX4ZAkCnTp2wdetW9O7dG8HBwahUqRJSUlJw9+5dbN26VTouxL9p1KgRFixYgPr166N9+/YIDw/HsmXL4OzsjBs3bkjrlSlTBq1atYKXlxfevXsHV1dXHD9+HPfv3weg2sU+a9YsBAcHo3z58ujZsyeKFCmCiIgIXLlyBYcPH/7qF/P3aNCgAYoVK4YFCxbA09MT1apVg4eHB2bOnIlr166hbt260NfXx4MHD7Bt2zYsWrQIrVu3xvr167F8+XK0aNEC+fLlQ0xMDFauXAkzM7M0wcLZ2RmVK1dGnz59kJiYCC8vL2TLlg0jRoyQ1pk7dy4aNGiAChUqoHv37oiPj8eSJUtgbm7+Q+c0mTJlCk6cOIFGjRohd+7cCA8Px/Lly+Hg4CAdQVQTr7U2pMdr9D1KlCgBd3d3+Pr6IjIyEtWqVcOFCxewfv16NG/eXBrMri4dHR2sWrUKDRo0QNGiRdG1a1fkzJkTz58/R3BwMMzMzLB3714AQOPGjeHn5wdzc3MUKVIEZ8+exeHDh5EtWzaV+xw+fDi2b9+OP//8E926dUOZMmUQERGBPXv2wMfHR2V3ryZ9z+dnkyZNUKNGDYwdOxahoaEoUaIEDh06hN27d2PQoEFSD1PJkiXRrl07LF++HFFRUahYsSKOHDny1WO8pNfnA2mAVuYapTMAYufOndL1ffv2CQDC1NRU5aKnpyfc3NyEEELcuXNHAPjXy8iRI6X71NfXFxUqVFB53P79+wtXV9df8hw1pUmTJsLIyEjl+Alf6tKli9DX15em7SmVSuHo6PjV6YGpkpKSxOzZs0XRokWFoaGhsLS0FGXKlBGTJ08WUVFR0nr4Ymrv51avXi3y588vDA0NRaFChcTatWulqbWfi4uLE56ensLKykpkyZJFNG/eXNy7d08AELNmzVJZ9/Xr18LT01M4OjoKfX19YWdnJ2rVqiV8fX3/82/1teOopFq3bl2aKYu+vr6iTJkywtjYWGTNmlUUL15cjBgxQrx48UIIIcSVK1dEu3btRK5cuYShoaGwtbUVjRs3FpcuXZLuI3Uq5Ny5c8X8+fOFo6OjMDQ0FFWqVBHXr19PU8fhw4dFpUqVhLGxsTAzMxNNmjQR//zzj8o6qX/DL6cdp04VTT3mypEjR0SzZs2Evb29MDAwEPb29qJdu3bi/v37Krf73tf6v/zI9OQvpw1/67l9baqwEJp5jb7lW4+ZnJwsJk+eLPLkySP09fWFo6OjGD16tEhISEjznL+1vX3L1atXRcuWLUW2bNmEoaGhyJ07t3BzcxNHjhyR1nn//r3o2rWrsLa2FlmyZBH16tUTd+/eTfM3FkKId+/eiX79+omcOXMKAwMD4eDgINzd3aXPgtTpydu2bVO53fdO4f3W6/ilr/0tYmJixODBg4W9vb3Q19cX+fPnF3PnzlWZni2EEPHx8WLAgAEiW7ZswtTUVDRp0kQ8ffo0zfRkIb7v84HTk389hRDp1IenRQqFAjt37kTz5s0BAFu2bEGHDh1w+/btNIO+smTJAjs7OyQlJf3nVLps2bJJg+xy586NOnXqYNWqVdJyb29vTJs2TaWLnrTj2rVrKFWqFP766y906NBB2+X8sNDQUOTJkwdz585VOfIvEVFmkSl2/ZQqVQopKSkIDw+X5vd/ycDAAIUKFfru+6xUqVKaAVn379//4cFw9OPi4+PTDIj08vKCjo4OqlatqqWqiIhIEzJMUImNjVXZ7/j48WNcu3YNVlZWKFCgADp06IDOnTtj/vz5KFWqFN68eYMjR47AxcXlhwawDR48GBUrVsSMGTPg5uaGCxcuwNfXF76+vpp8WvQd5syZg8uXL6NGjRrQ09PDgQMHcODAAfTq1Svdp8cSEVE60/a+J01J3Vf65SV1n2tSUpKYMGGCcHJyEvr6+iJHjhyiRYsW4saNGz/8mHv37hXFihWTxlB8zzgH0rxDhw6JSpUqCUtLS6Gvry/y5csnJk2aJJKTk7Vd2k/7fIwKEVFmlCHHqBAREVHGkGmOo0JERES/HwYVIiIikq3fejCtUqnEixcvkDVr1t/q8N1ERESZmRACMTExsLe3h47Ov/eZ/NZB5cWLF5zVQURE9Jt6+vQpHBwc/nWd3zqopJ6M6unTpzAzM9NyNURERPQ9oqOj4ejoqHJSyW/5rYNK6u4eMzMzBhUiIqLfzPcM2+BgWiIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki09bRdARETy5TRqv7ZLIC0LndVIq4/PHhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki3ZBJVZs2ZBoVBg0KBB2i6FiIiIZEIWQeXixYtYsWIFXFxctF0KERERyYjWg0psbCw6dOiAlStXwtLSUtvlEBERkYxoPah4enqiUaNGqF279n+um5iYiOjoaJULERERZVx62nzwzZs348qVK7h48eJ3rT9z5kxMnjw5nasiIiIiudBaj8rTp08xcOBA+Pv7w8jI6LtuM3r0aERFRUmXp0+fpnOVREREpE1a61G5fPkywsPDUbp0aaktJSUFJ06cwNKlS5GYmAhdXV2V2xgaGsLQ0PBXl0pERERaorWgUqtWLdy8eVOlrWvXrihUqBBGjhyZJqQQERFR5qO1oJI1a1YUK1ZMpc3U1BTZsmVL005ERESZk9Zn/RARERF9i1Zn/Xzp2LFj2i6BiIiIZIQ9KkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWz8UVB4+fIhx48ahXbt2CA8PBwAcOHAAt2/f1mhxRERElLmpHVSOHz+O4sWL4/z58wgICEBsbCwA4Pr165g4caLGCyQiIqLMS+2gMmrUKEybNg1BQUEwMDCQ2mvWrIlz585ptDgiIiLK3NQOKjdv3kSLFi3StNva2uLt27caKYqIiIgI+IGgYmFhgZcvX6Zpv3r1KnLmzKmRooiIiIiAHwgqbdu2xciRI/Hq1SsoFAoolUqcPn0aw4YNQ+fOndOjRiIiIsqk1A4qM2bMQKFCheDo6IjY2FgUKVIEVatWRcWKFTFu3Lj0qJGIiIgyKT11b2BgYICVK1di/PjxuHXrFmJjY1GqVCnkz58/PeojIiKiTEztoJIqV65cyJUrlyZrISIiIlKhdlARQmD79u0IDg5GeHg4lEqlyvKAgACNFUdERESZm9pBZdCgQVixYgVq1KiB7NmzQ6FQpEddREREROoHFT8/PwQEBKBhw4bpUQ8RERGRRO1ZP+bm5sibN2961EJERESkQu2gMmnSJEyePBnx8fHpUQ8RERGRRO1dP25ubti0aRNsbW3h5OQEfX19leVXrlzRWHFERESUuakdVNzd3XH58mV07NiRg2mJiIgoXakdVPbv34/AwEBUrlw5PeohIiIikqg9RsXR0RFmZmbpUQsRERGRCrWDyvz58zFixAiEhoamQzlERERE/6P2rp+OHTviw4cPyJcvH0xMTNIMpo2IiNBYcUSZndOo/dougbQsdFYjbZdApFVqBxUvL690KIOIiIgorR+a9UNERET0K3xXUImOjpYG0EZHR//ruhxoS0RERJryXUHF0tISL1++hK2tLSwsLL567BQhBBQKBVJSUjReJBEREWVO3xVUjh49CisrKwBAcHBwuhZERERElOq7gkq1atWQN29eXLx4EdWqVUvvmoiIiIgAqHEcldDQUO7WISIiol9K7QO+aZK3tzdcXFxgZmYGMzMzVKhQAQcOHNBmSURERCQjak1PDgwMhLm5+b+u07Rp0+++PwcHB8yaNQv58+eHEALr169Hs2bNcPXqVRQtWlSd0oiIiCgDUiuo/NcxVNSd9dOkSROV69OnT4e3tzfOnTvHoEJERETqBZVXr17B1tY2XQpJSUnBtm3bEBcXhwoVKnx1ncTERCQmJkrX/+uYLkRERPR7++4xKl87doom3Lx5E1myZIGhoSF69+6NnTt3okiRIl9dd+bMmTA3N5cujo6O6VITERERycN3BxUhRLoUULBgQVy7dg3nz59Hnz594O7ujn/++eer644ePRpRUVHS5enTp+lSExEREcnDd+/6cXd3h7GxscYLMDAwgLOzMwCgTJkyuHjxIhYtWoQVK1akWdfQ0BCGhoYar4GIiIjk6buDytq1a9OzDolSqVQZh0JERESZl9pnT9ak0aNHo0GDBsiVKxdiYmKwceNGHDt2DIGBgdosi4iIiGRCq0ElPDwcnTt3xsuXL2Fubg4XFxcEBgaiTp062iyLiIiIZEKrQWX16tXafHgiIiKSuR8+hH5ISAgCAwMRHx8PIP1mBREREVHmpXZQeffuHWrXro0CBQqgYcOGePnyJQCge/fuGDp0qMYLJCIiosxL7aAyePBg6Onp4cmTJzAxMZHa27Rpg4MHD2q0OCIiIsrc1B6jcujQIQQGBsLBwUGlPX/+/AgLC9NYYURERERq96jExcWp9KSkioiI4MHYiIiISKPUDipVqlTBhg0bpOsKhQJKpRJz5sxBjRo1NFocERERZW5q7/qZM2cOatWqhUuXLiEpKQkjRozA7du3ERERgdOnT6dHjURERJRJqd2jUqxYMdy/fx+VK1dGs2bNEBcXh5YtW+Lq1avIly9fetRIREREmdQPHfDN3NwcY8eO1XQtRERERCrU7lE5ePAgTp06JV1ftmwZSpYsifbt2+P9+/caLY6IiIgyN7WDyvDhwxEdHQ0AuHnzJoYMGYKGDRvi8ePHGDJkiMYLJCIiosxL7V0/jx8/RpEiRQAAO3bsQJMmTTBjxgxcuXIFDRs21HiBRERElHmp3aNiYGCADx8+AAAOHz6MunXrAgCsrKyknhYiIiIiTVC7R6Vy5coYMmQIKlWqhAsXLmDLli0AgPv376c5Wi0RERHRz1C7R2Xp0qXQ09PD9u3b4e3tjZw5cwIADhw4gPr162u8QCIiIsq81O5RyZUrF/bt25emfeHChRopiIiIiCjVDx1HRalUIiQkBOHh4VAqlSrLqlatqpHCiIiIiNQOKufOnUP79u0RFhYGIYTKMoVCgZSUFI0VR0RERJmb2kGld+/eKFu2LPbv348cOXJAoVCkR11ERERE6geVBw8eYPv27XB2dk6PeoiIiIgkas/6KV++PEJCQtKjFiIiIiIVaveo9O/fH0OHDsWrV69QvHhx6Ovrqyx3cXHRWHFERESUuakdVFq1agUA6Natm9SmUCgghOBgWiIiItKoHzrXDxEREdGvoHZQyZ07d3rUQURERJTGDx3w7eHDh/Dy8sKdO3cAAEWKFMHAgQORL18+jRZHREREmZvaQSUwMBBNmzZFyZIlUalSJQDA6dOnUbRoUezduxd16tTReJHa4jRqv7ZLIC0LndVI2yUQEWVqageVUaNGYfDgwZg1a1aa9pEjR2aooEJERETapfZxVO7cuYPu3bunae/WrRv++ecfjRRFREREBPxAULGxscG1a9fStF+7dg22traaqImIiIgIwA/s+unZsyd69eqFR48eoWLFigA+jVGZPXs2hgwZovECiYiIKPNSO6iMHz8eWbNmxfz58zF69GgAgL29PSZNmoQBAwZovEAiIiLKvNQOKgqFAoMHD8bgwYMRExMDAMiaNavGCyMiIiL6oeOoAEB4eDju3bsHAChUqBBsbGw0VhQRERER8AODaWNiYtCpUyfY29ujWrVqqFatGuzt7dGxY0dERUWlR41ERESUSakdVHr06IHz589j//79iIyMRGRkJPbt24dLly7Bw8MjPWokIiKiTErtXT/79u1DYGAgKleuLLXVq1cPK1euRP369TVaHBEREWVuaveoZMuWDebm5mnazc3NYWlpqZGiiIiIiIAfCCrjxo3DkCFD8OrVK6nt1atXGD58OMaPH6/R4oiIiChzU3vXj7e3N0JCQpArVy7kypULAPDkyRMYGhrizZs3WLFihbTulStXNFcpERERZTpqB5XmzZunQxlEREREaakdVCZOnJgedRARERGlofYYladPn+LZs2fS9QsXLmDQoEHw9fXVaGFEREREageV9u3bIzg4GMCnQbS1a9fGhQsXMHbsWEyZMkXjBRIREVHmpXZQuXXrFv744w8AwNatW1G8eHGcOXMG/v7+WLdunabrIyIiokxM7aCSnJwMQ0NDAMDhw4fRtGlTAJ/O9/Py5UvNVkdERESZmtpBpWjRovDx8cHJkycRFBQkHY32xYsXyJYtm8YLJCIiosxL7aAye/ZsrFixAtWrV0e7du1QokQJAMCePXukXUJEREREmqD29OTq1avj7du3iI6OVjlkfq9evWBiYqLR4oiIiChzU7tHBQCEELh8+TJWrFiBmJgYAICBgQGDChEREWmU2j0qYWFhqF+/Pp48eYLExETUqVMHWbNmxezZs5GYmAgfH5/0qJOIiIgyIbV7VAYOHIiyZcvi/fv3MDY2ltpbtGiBI0eOaLQ4IiIiytzU7lE5efIkzpw5AwMDA5V2JycnPH/+XGOFEREREando6JUKpGSkpKm/dmzZ8iaNatGiiIiIiICfiCo1K1bF15eXtJ1hUKB2NhYTJw4EQ0bNtRkbURERJTJqb3rZ/78+ahXrx6KFCmChIQEtG/fHg8ePIC1tTU2bdqUHjUSERFRJqV2UHFwcMD169exZcsWXL9+HbGxsejevTs6dOigMriWiIiI6GepHVQAQE9PDx06dECHDh2ktpcvX2L48OFYunSpxoojIiKizE2toHL79m0EBwfDwMAAbm5usLCwwNu3bzF9+nT4+Pggb9686VUnERERZULfPZh2z549KFWqFAYMGIDevXujbNmyCA4ORuHChXHnzh3s3LkTt2/fTs9aiYiIKJP57qAybdo0eHp6Ijo6GgsWLMCjR48wYMAA/P333zh48KB0FmUiIiIiTfnuoHLv3j14enoiS5Ys6N+/P3R0dLBw4UKUK1cuPesjIiKiTOy7g0pMTAzMzMwAALq6ujA2NuaYFCIiIkpXag2mDQwMhLm5OYBPR6g9cuQIbt26pbJO06ZNNVcdERERZWpqBRV3d3eV6x4eHirXFQrFVw+vT0RERPQjvjuoKJXK9KyDiIiIKA21z/VDRERE9KtoNajMnDkT5cqVQ9asWWFra4vmzZvj3r172iyJiIiIZESrQeX48ePw9PTEuXPnEBQUhOTkZNStWxdxcXHaLIuIiIhk4ofO9aMpBw8eVLm+bt062Nra4vLly6hataqWqiIiIiK50GpQ+VJUVBQAwMrK6qvLExMTkZiYKF2Pjo7+JXURERGRdvzQrp/IyEisWrUKo0ePRkREBADgypUreP78+Q8XolQqMWjQIFSqVAnFihX76jozZ86Eubm5dHF0dPzhxyMiIiL5Uzuo3LhxAwUKFMDs2bMxb948REZGAgACAgIwevToHy7E09MTt27dwubNm7+5zujRoxEVFSVdnj59+sOPR0RERPKndlAZMmQIunTpggcPHsDIyEhqb9iwIU6cOPFDRfTr1w/79u1DcHAwHBwcvrmeoaEhzMzMVC5ERESUcak9RuXixYtYsWJFmvacOXPi1atXat2XEAL9+/fHzp07cezYMeTJk0fdcoiIiCgDUzuoGBoafnUQ6/3792FjY6PWfXl6emLjxo3YvXs3smbNKgUdc3NzGBsbq1saERERZTBq7/pp2rQppkyZguTkZACfzu/z5MkTjBw5Eq1atVLrvry9vREVFYXq1asjR44c0mXLli3qlkVEREQZkNpBZf78+YiNjYWtrS3i4+NRrVo1ODs7I2vWrJg+fbpa9yWE+OqlS5cu6pZFREREGZDau37Mzc0RFBSEU6dO4caNG4iNjUXp0qVRu3bt9KiPiIiIMrEfPuBb5cqVUblyZU3WQkRERKRC7aCyePHir7YrFAoYGRnB2dkZVatWha6u7k8XR0RERJmb2kFl4cKFePPmDT58+ABLS0sAwPv372FiYoIsWbIgPDwcefPmRXBwMI8cS0RERD9F7cG0M2bMQLly5fDgwQO8e/cO7969w/3791G+fHksWrQIT548gZ2dHQYPHpwe9RIREVEmonaPyrhx47Bjxw7ky5dPanN2dsa8efPQqlUrPHr0CHPmzFF7qjIRERHRl9TuUXn58iU+fvyYpv3jx4/SAdvs7e0RExPz89URERFRpqZ2UKlRowY8PDxw9epVqe3q1avo06cPatasCQC4efMmD4dPREREP03toLJ69WpYWVmhTJkyMDQ0hKGhIcqWLQsrKyusXr0aAJAlSxbMnz9f48USERFR5qL2GBU7OzsEBQXh7t27uH//PgCgYMGCKFiwoLROjRo1NFchERERZVo/fMC3QoUKoVChQpqshYiIiEjFDwWVZ8+eYc+ePXjy5AmSkpJUli1YsEAjhRERERGpHVSOHDmCpk2bIm/evLh79y6KFSuG0NBQCCFQunTp9KiRiIiIMim1B9OOHj0aw4YNw82bN2FkZIQdO3bg6dOnqFatGv7888/0qJGIiIgyKbWDyp07d9C5c2cAgJ6eHuLj45ElSxZMmTIFs2fP1niBRERElHmpHVRMTU2lcSk5cuTAw4cPpWVv377VXGVERESU6ak9RsXV1RWnTp1C4cKF0bBhQwwdOhQ3b95EQEAAXF1d06NGIiIiyqTUDioLFixAbGwsAGDy5MmIjY3Fli1bkD9/fs74ISIiIo1SK6ikpKTg2bNncHFxAfBpN5CPj0+6FEZERESk1hgVXV1d1K1bF+/fv0+veoiIiIgkag+mLVasGB49epQetRARERGpUDuoTJs2DcOGDcO+ffvw8uVLREdHq1yIiIiINEXtwbQNGzYEADRt2hQKhUJqF0JAoVAgJSVFc9URERFRpqZ2UAkODk6POoiIiIjSUDuoVKtWLT3qICIiIkpD7TEqAHDy5El07NgRFStWxPPnzwEAfn5+OHXqlEaLIyIiosxN7aCyY8cO1KtXD8bGxrhy5QoSExMBAFFRUZgxY4bGCyQiIqLM64dm/fj4+GDlypXQ19eX2itVqoQrV65otDgiIiLK3NQOKvfu3UPVqlXTtJubmyMyMlITNREREREB+IGgYmdnh5CQkDTtp06dQt68eTVSFBERERHwA0GlZ8+eGDhwIM6fPw+FQoEXL17A398fw4YNQ58+fdKjRiIiIsqk1J6ePGrUKCiVStSqVQsfPnxA1apVYWhoiGHDhqF///7pUSMRERFlUmoHFYVCgbFjx2L48OEICQlBbGwsihQpgixZsqRHfURERJSJqb3r56+//sKHDx9gYGCAIkWK4I8//mBIISIionShdlAZPHgwbG1t0b59e/z99988tw8RERGlG7WDysuXL7F582YoFAq4ubkhR44c8PT0xJkzZ9KjPiIiIsrE1A4qenp6aNy4Mfz9/REeHo6FCxciNDQUNWrUQL58+dKjRiIiIsqk1B5M+zkTExPUq1cP79+/R1hYGO7cuaOpuoiIiIh+7KSEHz58gL+/Pxo2bIicOXPCy8sLLVq0wO3btzVdHxEREWViaveotG3bFvv27YOJiQnc3Nwwfvx4VKhQIT1qIyIiokxO7aCiq6uLrVu3ol69etDV1VVZduvWLRQrVkxjxREREVHmpnZQ8ff3V7keExODTZs2YdWqVbh8+TKnKxMREZHG/NAYFQA4ceIE3N3dkSNHDsybNw81a9bEuXPnNFkbERERZXJq9ai8evUK69atw+rVqxEdHQ03NzckJiZi165dKFKkSHrVSERERJnUd/eoNGnSBAULFsSNGzfg5eWFFy9eYMmSJelZGxEREWVy392jcuDAAQwYMAB9+vRB/vz507MmIiIiIgBq9KicOnUKMTExKFOmDMqXL4+lS5fi7du36VkbERERZXLfHVRcXV2xcuVKvHz5Eh4eHti8eTPs7e2hVCoRFBSEmJiY9KyTiIiIMiG1Z/2YmpqiW7duOHXqFG7evImhQ4di1qxZsLW1RdOmTdOjRiIiIsqkfnh6MgAULFgQc+bMwbNnz7Bp0yZN1UREREQE4CeDSipdXV00b94ce/bs0cTdEREREQHQUFAhIiIiSg8MKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbWg0qJ06cQJMmTWBvbw+FQoFdu3ZpsxwiIiKSGa0Glbi4OJQoUQLLli3TZhlEREQkU3rafPAGDRqgQYMG2iyBiIiIZEyrQUVdiYmJSExMlK5HR0drsRoiIiJKb7/VYNqZM2fC3Nxcujg6Omq7JCIiIkpHv1VQGT16NKKioqTL06dPtV0SERERpaPfatePoaEhDA0NtV0GERER/SK/VY8KERERZS5a7VGJjY1FSEiIdP3x48e4du0arKyskCtXLi1WRkRERHKg1aBy6dIl1KhRQ7o+ZMgQAIC7uzvWrVunpaqIiIhILrQaVKpXrw4hhDZLICIiIhnjGBUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLVkElWXLlsHJyQlGRkYoX748Lly4oO2SiIiISAa0HlS2bNmCIUOGYOLEibhy5QpKlCiBevXqITw8XNulERERkZZpPagsWLAAPXv2RNeuXVGkSBH4+PjAxMQEa9as0XZpREREpGVaDSpJSUm4fPkyateuLbXp6Oigdu3aOHv2rBYrIyIiIjnQ0+aDv337FikpKciePbtKe/bs2XH37t006ycmJiIxMVG6HhUVBQCIjo5Ol/qUiR/S5X7p95Fe29b34jZI3AZJ29JjG0y9TyHEf66r1aCirpkzZ2Ly5Mlp2h0dHbVQDWUG5l7aroAyO26DpG3puQ3GxMTA3Nz8X9fRalCxtraGrq4uXr9+rdL++vVr2NnZpVl/9OjRGDJkiHRdqVQiIiIC2bJlg0KhSPd6M5Po6Gg4Ojri6dOnMDMz03Y5lAlxGyRt4zaYfoQQiImJgb29/X+uq9WgYmBggDJlyuDIkSNo3rw5gE/h48iRI+jXr1+a9Q0NDWFoaKjSZmFh8QsqzbzMzMz4BiWt4jZI2sZtMH38V09KKq3v+hkyZAjc3d1RtmxZ/PHHH/Dy8kJcXBy6du2q7dKIiIhIy7QeVNq0aYM3b95gwoQJePXqFUqWLImDBw+mGWBLREREmY/WgwoA9OvX76u7ekh7DA0NMXHixDS72oh+FW6DpG3cBuVBIb5nbhARERGRFmj9yLRERERE38KgQkRERLLFoEJERESyxaBCREREssWgQkRERLLFoEJERESyxaBCREREssWgQkRERLLFoEJERESyxaBCvyWlUqntEoiI6BdgUKHfko7Op0337du3AACeCYJ+tS/DMrdB0oYvt8OM+COOQYV+W4sWLULz5s3x8OFDKBQKbZdDmYyOjg6ioqIQGBgIANwGSSt0dHQQGRmJuXPn4v3799KPuIwk4z0jyrC+/MWqr68PY2NjGBgYaKkiysyUSiXmz58PDw8P7Nu3T9vlUCZ26NAhLFiwAEuXLtV2KemCZ0+m3050dDTMzMwAAFFRUTA3N9dyRZRZKJVKlV+sd+7cwerVqzF79mzo6upqsTLKTFJSUlS2t+TkZGzZsgXt2rXLkNshgwr9VgYPHoyUlBSMHj0aOXLk0HY5lAlFRkYiMjISjo6OKl8KX355EP2ML0Pxl969e4fTp0+jYsWKsLa2ltoz4nbIXT8ka1/maAcHB2zYsCHDvRHp9yCEwKhRo1C+fHmEhoaqLOM2ST/j5cuXePHiBd68eQPg09iTf+tH2Lp1K5o3b47jx4+rtGfE7ZA9KiQbqb8EhBBQKBTf/EXx/v17WFpaaqFCymj+61fr19YJCwvDuHHjsG7dugz5pUC/3tq1a7Fs2TI8ffoU+fLlQ+XKlTFnzhyVdb7WU+Ll5YV+/fpBT0/vV5b7yzGokFakhhHg0xtQCAE9PT08f/4cO3fuRNeuXWFqagrg0+4eS0tLTJgwIc1tiX7U5wHk6NGjePLkCZydnZE3b17Y29urrBMVFQWlUpkmIGfEbnb6tfbt2wc3NzcsX74cJiYmePToEebMmYOKFSti/fr1yJYtm/SZ9/btW4SEhMDV1VXlPj5+/Jihwwp3/dAvkZqHo6OjER8fD4VCgUOHDiEkJAS6urrQ09NDWFgYSpUqhRcvXkghJS4uDvr6+li4cCEiIiIYUkgjhBBSSBk1ahS6dOmCefPmoVevXhg2bBguXrwI4FP3e2JiIiZMmIDSpUvj3bt3KvfDkEI/6+LFi2jUqBG6dOkCNzc3jBgxAoGBgbhx4wY6dOgA4NPU9+TkZPj5+aFixYo4deqUyn1k5JACMKjQL/Tq1SsUL14cx48fx8aNG1G/fn38888/AD7tzilatChatGiB6dOnS7cxNTXFiBEj8ODBA1hZWTGkkEakbkfz5s3DX3/9hU2bNuHWrVto2bIl9u7di3HjxuHs2bMAAAMDA5QqVQq1atWChYWFFqumjOjx48d4+fKlSlu5cuWwZ88eXL58GT179gTw6XAMjRs3xvTp09P0qGR4gugX6tq1qzAzMxM6Ojpi5cqVUntSUpLYsmWLSElJkdqUSqU2SqRM4vXr16Jly5ZizZo1Qggh9uzZI8zMzETv3r1FqVKlRK1atcS5c+eEEKrb4sePH7VSL2VMgYGBInv27GLz5s1SW+r25u/vL5ydncXFixfT3C45OfmX1aht7FGhXyL1sM6enp6IiYmBgYEB7OzskJCQAODTrwU3NzeVQYvsPaH0ZGtrixEjRqB+/fq4evUqPD09MW3aNHh7e6NVq1Y4d+4cPD09cfnyZZVtkbt7SJMKFy6M6tWrw8/PD0eOHAHwv8++kiVLIjw8XDpVyOcy+u6ezzGo0C+RGkAcHR1x6tQpuLu7o23btti9ezfi4+PTrJ8Rz1dB2vOt7alUqVLIkSMHDhw4ABcXF/Tq1QsAYGVlBVdXVzRp0gSlSpX6laVSJuPo6IjevXsjMjISCxcuxJ49e6RlOXLkQJ48ebRYnTxknkhGWiH+f/Dry5cvkZycjFy5csHW1hYVK1ZEQkICunfvjnXr1qFx48YwMjKCj48PateuDWdnZ22XThmE+Gzg7KpVqxAeHg4DAwMMGzZMOv1CYmIinj9/jtDQUBQsWBCHDh1C06ZN0b9//3+dKk/0M1JnjVWvXh3Lly/HmDFjMHLkSAQGBsLFxQVbt26FQqFAnTp1tF2qVnF6MqW7gIAATJo0Ca9fv0ajRo3QokULNGnSBADQtWtX7Ny5E0OHDsXr16/h7e2NmzdvokiRIlqumjKaiRMnwsvLC+XKlcOFCxdQvnx5+Pn5wc7ODnv37sW0adPw/v176OvrQwiBGzduQE9PjzPNKF2kblcBAQFYvnw5Dh06hLt37yI4OBhLly6Fo6MjLCws4O/vD319/Uw9FZ5BhdLV7du3Ua9ePQwePBgmJibYtGkTDA0N4e7ujo4dOwIABg4ciCtXriAxMRG+vr4oWbKkdoumDOHzXpCPHz/C3d0d/fv3R6lSpRAaGopGjRrBzs4OO3fuhI2NDfbv34+QkBDExsZi5MiR0NPTy9RfDqQZqYFEfHHsKF1dXQQEBKBz585YsGCBtNsR+LS96ujoqGy/mWlMypcYVCjd3L17F9u2bUN8fDxmzJgBALh58yYmTJiA6OhodO3aVQorr169gqmpKbJmzarNkimD+Dyk3LlzB9HR0VixYgUmTJgAJycnAJ+mhdapUwfZs2fHrl27YGNjo3IfDCn0sz7fDt++fQuFQoFs2bIB+PSZV7p0aUyYMAG9e/eWbvNlDx579BhUKB0IIfD+/Xs0btwY//zzD5o0aQI/Pz9p+Y0bNzBhwgTEx8ejbdu26Nq1qxarpYxs+PDhUtf569evERAQgAYNGkgf/I8fP0aDBg0ghMDp06dVTu5G9DM+DxhTp07Frl27EB0dDWtra0yfPh01a9bE8+fPkTNnTi1XKn8cHUYap1AoYGVlhZkzZ6Jo0aK4cuUKgoKCpOUuLi6YOnUqkpOTpTcvkSZ8Prtn3759OHjwIBYvXozly5cjT548GDt2LK5fvy4dKTlPnjzYt28fSpYsyfNHkUalhpQpU6Zg0aJF0vR3a2trdOjQAevXr0/Ti0dfxx4V0ohvdU8eP34cY8aMgZ2dHTw9PVGzZk1p2e3bt2Fubg4HB4dfWSplAgEBAThz5gyyZcuG0aNHAwBiY2NRunRpmJmZYdWqVShRokSabZa7e0iT3r17h7p168LT0xPdunWT2nv16oW9e/ciODgYhQoV4u6d/8AeFfppqW+yM2fOYMGCBRg/fjxOnz6N5ORkVKtWDVOmTMGrV6+wdOlSHDt2TLpd0aJFGVJI4+Lj4zF+/HgsWLAAt2/fltqzZMmCK1euICYmBh4eHtL5fD7HkEKa9PHjR7x9+1bqrUs9wKWvry/s7e2xcOFCADy45X9hUKGf8vkUuwYNGuD06dPYs2cPxowZg+nTpyMpKQm1atXClClT8O7dO0ydOhUnT57UdtmUgRkbG+PkyZOoXbs2Ll++jD179iAlJQXA/8LK3bt3sWLFCi1XShnJ13ZOZM+eHXZ2dlizZg0AwMjICElJSQAAZ2dnBpTvxKBCPyW1J2XAgAFYsGABduzYgW3btuHy5cvYsmULxo0bJ4WVUaNGQV9fn0daJI35fEyKEEL6srCyssLGjRthaWmJuXPnIjAwUFpmamqKV69ewdfXVys1U8ajVCql0PHixQuEh4fjw4cPAIBJkybh7t270sye1IMMPnv2jCe5/E4co0I/JPWNqVAosHz5cly7dg2+vr54/PgxateujcqVK8PMzAzbtm2Dh4cHxowZA0NDQ3z48AEmJibaLp8ygM+nfi5ZsgTXr1/Ho0ePMGjQIJQuXRoODg548+YNmjVrBl1dXYwZMwb16tVTOcIsx6TQz/D394erqyvy5csHABg9ejQCAwMRFhaG2rVro2nTpujQoQNWrlyJqVOnIlu2bChWrBgePnyIyMhI6aCC9O8YVOi7pH4pfB40rl27hpIlSyI6OhpPnz6Fs7Mz6tevjzx58mDNmjWIioqSjjDbpUsXTJ8+nYPG6Kd9uQ2NHj0aq1evRq9evfDs2TOcPXsWzZo1Q69eveDs7Iw3b96gZcuWePPmDdatWwdXV1ctVk8ZxYEDB9C4cWOMHDkSgwYNwoEDBzBixAh4eXnh3bt3uHLlCgIDAzF+/Hj07t0bN2/ehJeXF3R0dGBpaYkZM2bwoILfK13PzUwZyqNHj0S7du3EP//8I7Zu3SoUCoW4cOGCdErymzdvikKFConz588LIYR4+PChaNy4sRgzZox48uSJNkunDCYlJUUIIYSfn5/IkyePuHz5shBCiJMnTwqFQiHy588vBg4cKB49eiSEEOLly5eiV69e4uPHj1qrmTKepUuXCgcHBzF16lTRr18/sXLlSmnZ06dPxZQpU4STk5M4ePDgV2+fnJz8q0r9rbHPib5bQkICTp48iS5duuDatWtYu3YtypUrJ+0GEkLg48ePOHv2LIoWLYoNGzYAAIYNG8ZjVNBP69SpE2xsbLBgwQLo6OggOTkZBgYG6N27N0qXLo1du3aha9euWLVqFV69eoVp06ZBR0cHPXv2ROHChaXBs/wFSz8rKSkJBgYG8PT0hImJCUaPHo2YmBhMmzZNWsfBwQGdO3fGoUOHcOnSJdSrVy/NyS252+c7aTsp0e8h9Resj4+P0NHRESVKlBBXr15VWScqKkp06dJF5MuXTzg5OQkbGxvply7Rz4iKihKTJ08WVlZWYtKkSVL78+fPxevXr8XLly9F2bJlxfz586X17e3tRY4cOcSiRYuEEELq+SPSlJkzZ4rw8HDh7+8vTExMRMOGDcX9+/dV1mnTpo1o2bKllirMGDjrh/6TEAI6OjoQQsDe3h7z58/Hx48fMW7cOJw6dUpaz8zMDPPmzcPy5csxceJEnD9/HqVLl9Zi5ZQRxMTEwMzMDH369MG4cePg5eWFiRMnAgDs7e1ha2uLly9f4v3799L4k+fPn6Nu3bqYMGECPD09AfBYFfTzxGdDOtevX4+pU6fiwYMHaN++PRYuXIgrV67Ax8cH9+7dAwBER0fj8ePHyJUrl7ZKzhDY70T/Svz/wMWjR4/i+PHjGDRoEJo0aYLatWvDzc0Ns2bNwpgxY1CxYkUAn046WLduXS1XTRnFiBEjsGLFCjx8+BA2Njbo2LEjhBCYOnUqAGDy5MkAPoUZXV1dnD59GkIIzJo1CyYmJtKUUO7uIU1IDbtHjhzB1atX4evrK3329erVC8nJyZg8eTIOHjyI0qVLIy4uDklJSZgzZ442y/79abM7h+Qttat8+/btwtzcXIwePVpcvHhRWn7jxg1RpEgR0bhxY/HXX3+JSZMmCYVCIZ4+fcpudtKI69evi6pVq4qCBQuKN2/eCCGECA8PF/PnzxcWFhZiwoQJ0rr9+vUT+fLlEw4ODsLV1VUkJSUJIbjLhzTr2LFjonjx4iJbtmxi165dQgghEhMTpeWrV68WWbJkEaVLlxYbNmyQBnBz4OyP4/Rk+lcXLlxA/fr1MXv2bPTs2VNqj46OhpmZGe7cuYOePXsiPj4eUVFR2Lp1K3f3kEacPXsWb968QZEiRdCmTRvExsZKZzh+8+YN/Pz8MHXqVOlkb8CnKfMKhQLFixeHjo4OPn78yAGL9FPEF9PhY2NjMXfuXPj6+qJ8+fLYtGkTjI2NkZycDH19fQDAggULcObMGWzbtg0KhYI9ej+JQYX+1dKlS7Fz504cOXIEUVFROHr0KP766y/cuXMHw4YNQ7du3RAeHo6oqCiYm5vD1tZW2yVTBtG5c2e8ePEChw8fRmhoKFq3bo2YmJg0YWXatGno168fpkyZonJ7fjmQJi1btgwODg5o1qwZ4uPjMW/ePOzcuRPVq1fHjBkzYGRkpBJWUgPOl0GH1MfBtPSv7OzscPnyZcycOROtW7fG2rVrYWRkhEaNGqFHjx64f/8+bG1tkT9/foYU0qhly5bh2bNnWLp0KZycnLBp0yaYm5ujUqVKePv2LWxsbNCpUydMmDAB06ZNw+rVq1Vuz5BCmvLmzRscPXoUffv2xcGDB2FsbIwhQ4agcePGOHPmDMaOHYuEhATo6+vj48ePAMCQokHsUSFJ6psqNjYWWbJkAQC8fv0aS5YswdatW1GzZk106dIFf/zxB16/fo2mTZti3bp1KFq0qJYrp4wmtTdk8eLFuHr1KhYsWABLS0vcvXsXnTt3RlRUlNSz8urVKxw/fhytWrXibh7SiC+PdwIA169fx+LFi3H48GH4+PigQYMGiIuLw5w5c3D48GEULlwYy5cvl87lQ5rDHhWSKBQK7N+/H+3atUP16tWxbt066OnpYdq0aTh//jx8fHzg6uoKHR0dLFmyBHFxcexFoXSR2htSvXp1nDhxAvv37wcAFCxYEH5+frC0tETVqlXx+vVr2NnZoU2bNtDT05N+zRL9jNSQ8urVK6mtRIkSGDhwIGrUqIHevXvj4MGDMDU1xYgRI/DHH39AR0dH2u1DGqalQbwkQ6dPnxZGRkZi+PDhon79+sLFxUV4eHiIkJAQaZ3g4GDRq1cvYWVlleaAb0Q/KvWAgl/j4+MjChQoIO7duye13bt3Tzg5OYm2bdv+ivIok/h8O9y8ebPImzevykxHIYS4du2aaNasmciVK5c4duyYEEKI+Ph4aXbZv23L9GPYo0IAgLCwMAQFBWH69OmYM2cODhw4gF69euHGjRuYOXMmHj16hLi4OJw9exbh4eE4fvw4SpYsqe2yKQP4vJv9woULOHPmDI4fPy4tb9q0KcqXL4/g4GCprUCBAjhx4gT++uuvX14vZUyJiYnSdpiUlIR8+fKhUKFC8PT0xOXLl6X1SpQogebNm+Pp06eoW7cuzpw5AyMjI2lMype7jOjn8S+aCS1duhR///23dP3evXto06YN1qxZAyMjI6nd09MTHTp0wO3btzFnzhxERkZi+PDhWL9+PYoVK6aN0imD+fyDfcyYMejSpQu6desGd3d3tGnTBtHR0ciRI4e0/z85OVm6raOjI3R1dZGSkqKt8imDOHDgAPz8/AAAPXv2RM2aNVG2bFkMHToUdnZ28PDwwKVLl6T1c+XKhbZt22L+/PkoX7681M6Bs+lE21069Gs9fvxYtG/fXjx48EClfdSoUcLW1la0bNlSOrBWKm9vb1GwYEExYMAAHrSI0sW8efNEtmzZxPnz50VKSoqYMWOGUCgU4tSpU9I6lSpVEh4eHlqskjKqdu3aCScnJ1GvXj1hbW0trl+/Li07evSoaN68uShWrJg4cOCAePz4sWjevLkYOnSotA7Pyp2+GFQyobi4OCGEEOfOnRPbt2+X2idMmCCKFy8uxo0bJ16/fq1ym5UrV4rHjx//yjIpk1AqlcLd3V34+voKIYTYsWOHsLCwED4+PkIIIWJiYoQQQhw4cEA0bdpU3LhxQ2u1UsZVsmRJoVAoVE56merkyZOiU6dOQqFQiAIFCggXFxfpRxuPfJz+OJcvEzI2NkZkZCRmzpyJ58+fQ1dXF82bN8fkyZORnJyM/fv3QwiBgQMHwsbGBgDQo0cPLVdNGVVCQgLOnz+P6tWr49ixY3B3d8fcuXPh4eGBjx8/Ys6cOahQoQJcXV0xZcoUXLhwAcWLF9d22ZRBJCUlISEhAc7OzsiVKxe2bNmCnDlzom3bttJhGipXrozy5cujZ8+eSE5ORrVq1aCrq8sjH/8iHKOSCSkUClhYWGDo0KHIkycPvLy8EBAQAACYMWMG6tevj6CgIMyYMQNv377VcrWUkdy4cQPPnj0DAAwePBjHjx+HsbEx2rdvj7/++gsNGzbEwoULpZMJvn//HpcuXcK9e/dgaWkJPz8/5M6dW5tPgTIYAwMDmJmZYdu2bdi9ezfKlSuHOXPmYPPmzYiJiZHWS0hIQJUqVVCzZk1pbBRDyq/BoJIJiU+7/FClShUMHjwYlpaWWLx4sUpYcXV1xdWrV1VOa070o4QQuH//PmrUqIE1a9agd+/eWLRoESwtLQEArq6uCAsLQ/ny5VGhQgUAwIsXL9ClSxdERkaiX79+AIB8+fKhdu3aWnselPEIIaBUKqXr69evR8WKFbFw4UJs2LABT548Qc2aNfHnn39K6wM88vGvxCPTZkKpR/2MioqCiYkJbty4genTp+P9+/cYOHAgmjdvDuDTYaNTd/0QacLKlSsxYsQIJCQkYPfu3ahbt650ROQtW7ZgypQpEEJAT08PxsbGUCqVOHPmDPT19XnuHvppERERsLKyUmlL3f62bduGoKAg+Pr6AgB69eqFY8eOISUlBVZWVjh9+jSPOqsl7FHJZD5+/AhdXV2EhoaievXqOHToEMqUKYNhw4bBxsYGkydPxr59+wCAIYU0JvUXq6OjIwwNDWFmZoZz584hNDRUmtLZpk0bbNiwAVOmTIGbmxtGjhyJc+fOSedPYUihn7Fo0SKUK1dOZXcOACmkdOnSBSVKlJDafX19sWLFCixZsgTnzp2DgYEBj3ysLdoZw0u/wrdGo4eEhIjs2bOLHj16qEyrO3bsmOjUqZMIDQ39VSVSBvflNpiUlCTi4+OFt7e3yJkzpxgzZsx/bm+c+kk/a8WKFcLQ0FBs3LgxzbInT56I4sWLi6VLl0ptX9vmuB1qD3f9ZFDi/7szz549izt37iAkJASdO3dGjhw5sH79ely6dAnr169Pc4bPhIQElYO+Ef2oz484GxERgZiYGJWBsF5eXpg3bx66d++Orl27wsnJCU2aNMHYsWPh6uqqrbIpg1m5ciX69+8PPz8//Pnnn4iMjERcXBwSEhJga2uLrFmz4sGDB8ifP7+2S6VvYFDJwHbs2IFevXpJJ2978+YN2rRpg5EjRyJr1qzaLo8ysM9DypQpU3Do0CHcunULbm5uaNGiBRo0aADgU1jx8vJCsWLF8O7dOzx58gShoaE8uRtpxKNHj+Ds7Aw3Nzds3rwZt27dQt++ffHmzRuEhYWhRo0a6NOnDxo3bqztUulfcG5VBnXr1i0MHjwY8+fPR5cuXRAdHQ0LCwsYGxszpFC6Sw0pEyZMgK+vL+bOnQsnJyf07t0bDx48QGRkJNq1a4dBgwbB2toa169fR0JCAk6ePCmdBZlTP+ln2djYYPbs2ZgwYQKGDRuGQ4cOoUqVKmjWrBmio6Oxfft2jBs3DtbW1uzFkzNt7ncizTh69Kh4+PBhmrYKFSoIIYS4c+eOyJ07t+jRo4e0/OHDh9znSunq6NGjomjRouLEiRNCCCHOnDkjDAwMRJEiRUT58uXFtm3bpHU/PzUDT9NAmpSQkCDmzZsndHR0RLdu3URSUpK07NKlS6JgwYJi2bJlWqyQ/gtn/fzGhBC4evUqGjRoAG9vb4SFhUnLnj9/DiEEYmNjUb9+fdStWxcrVqwAAAQFBcHb2xvv37/XVumUAYkv9iLnzJkTffr0QZUqVXDo0CE0btwYvr6+CAoKwsOHD7F48WKsXr0aAFR6T9iTQppkaGiI3r17Y8eOHejRowf09fWlbbVMmTIwMjLC06dPtVwl/RsGld+YQqFAqVKlMH/+fGzduhXe3t549OgRAKBRo0Z4/fo1zMzM0KhRI/j6+krd8YGBgbhx4wane5LGKJVKaUD2o0ePEBcXh/z586Ndu3ZISEjAokWLMGDAAHTq1An29vYoWrQoQkJCcOfOHS1XTpmBqakpGjRoIB1MMHVbDQ8Ph7GxMYoWLarN8ug/8KfLbyx1P76npycAYO7cudDV1UWPHj2QJ08ejB8/HjNmzMDHjx/x4cMHhISEYNOmTVi1ahVOnTolHRWU6Gd8PnB2woQJOHv2LIYPH44aNWrAysoKcXFxePnyJUxMTKCjo4PExEQ4OTlhxIgRqF+/vparp4xIfDaTMZWhoaH0/5SUFLx9+xY9e/aEQqFAu3btfnWJpAYGld9Yao/IoUOHoKOjg+TkZHh5eSEhIQEjR46Em5sb4uPjMWPGDGzfvh3Zs2eHgYEBgoODUaxYMS1XTxnF5yFlxYoV8PX1RalSpaSZO4mJibCyssKpU6ekAbPv3r3DmjVroKOjoxJ0iH5EWFgYIiIikC1bNtjZ2f3rEWSTk5Ph5+eHTZs2ISIiAufOnZPO3cNeZnni9OTfXGBgoHQiN1NTUzx48ACLFy9G3759MXLkSNjY2CAmJgbHjx+Hk5MTbG1tYWtrq+2y6Tf3Zbi4f/8+mjdvjtmzZ6NJkyZp1rt48SLGjRuH2NhYWFlZISAgAPr6+gwp9NM2bNiA+fPnIzw8HNbW1ujfv7/UU5Lqy+0sKCgIt2/fRr9+/TjL7DfAoPIbUyqV6NChAxQKBTZu3Ci1L1myBCNGjICnpyf69u2LvHnzarFKymhatmyJMWPGoGzZslLbtWvXUL9+fRw/fhwFCxb86kEEExISIISAkZERFAoFvxzop23YsAGenp7S4fFnzJiBR48e4fTp09K2lRpSIiMjcejQIbi5uancB3tS5I8/ZX5jqb8QUrvYk5KSAAD9+/eHh4cH1q5di8WLF6vMBiL6Webm5nBxcVFpMzIywvv373Hr1i2pLfX8PmfPnsWOHTugo6MDY2NjKBQKKJVKhhT6KZcuXcLUqVOxdOlSdOvWDcWLF8fgwYPh7OyMM2fO4Pbt24iOjpZ2i69fvx59+/bFX3/9pXI/DCnyx6DyG3rx4oX0/4IFC2Lv3r0IDw+HgYEBkpOTAQAODg4wMTFBcHAwjI2NtVUqZSDPnz8HAKxduxYGBgZYvHgxDh06hKSkJDg7O6NNmzaYO3cuDh8+DIVCAR0dHaSkpGD69OkIDg5WGTfA3T30sxITEzFo0CA0atRIaps0aRKOHDmCdu3aoXPnzmjbti0iIiKgr6+Phg0bYtiwYRw4+xvirp/fzPXr19GvXz+0b98effr0QVJSEmrWrIm3b9/i2LFjsLOzAwCMHDkSRYsWRePGjdOc1pxIXT179gQAjB49WtqV6OLigrdv32Lz5s2oWrUqTp48iYULF+LmzZvo0KEDDAwMcOTIEbx58wZXrlxhDwpplFKpxJs3b5A9e3YAQOfOnXH48GHs2bMHjo6OOH78OKZNm4aRI0eiffv2KmNWuLvn98KfNb8ZExMTWFhYYPv27Vi3bh0MDAywYsUK2NjYoHDhwmjevDnq1q2LRYsWoWzZsgwppBEuLi44ePAgvL29ERISAgC4ceMGChYsiA4dOuDEiROoUqUKpkyZgs6dO8PPzw9Hjx5Frly5cPnyZWnAIpGm6OjoSCEFAIYNG4bz58+jbNmyyJ49Oxo0aICIiAi8fv06zVRlhpTfC3tUfkMhISEYM2YMXr16hZ49e6JTp05ISUnBvHnzEBYWBiEE+vfvjyJFimi7VMpA1qxZgwkTJqBt27bo2bMnChYsCACoWrUqHj9+DH9/f1StWhUA8OHDB5iYmEi35cBZ+tWePXuGjh07YtiwYTzp4G+OQeU3cOXKFbx8+VJlX2xISAjGjRuH0NBQ9O/fHx06dNBihZSRfT61c/Xq1ZgwYQLatWuXJqyEhYVhw4YNqFChgsp4lK8dfItIHZ9vQ6n/T/33zZs3sLGxUVk/Li4O7dq1Q1RUFI4ePcoelN8cg4rMxcTEoFGjRtDV1cWIESPQoEEDaVloaCjq168PExMT9OjRA3379tVipZTRfOsYJytXrsTkyZPRpk0b9OrVSworNWvWxOnTp3Hu3DmUKlXqV5dLGdTXtsPUtoCAAGzatAmLFi2Cvb094uPjsXv3bvj5+eH58+e4ePEi9PX1OSblN8cxKjKVmh+zZs2KOXPmQE9PD0uXLsX+/fuldZycnFCjRg28evUKR44cQWRkpJaqpYzm8y+HM2fOIDg4GNevXwfwaWDt+PHjsXnzZvj6+uLevXsAgKNHj6JHjx5ppi4T/ahTp05JJwwcMmQIZs2aBeDT+JQtW7agc+fOqF27Nuzt7QF8OqHl48ePkTdvXly6dAn6+vr4+PEjQ8pvjj0qMpPanZn6CyD1C+P8+fMYNWoUTE1N0adPH2k30NChQ5E3b160bNkSOXLk0HL1lBF83s0+ZMgQbNmyBbGxsXBwcECuXLlw4MABAMCKFSswbdo0tG3bFu7u7iqnZeAvWPoZQghERUXB1tYWDRo0gLW1NQICAnDy5EkUK1YMkZGRcHV1haenJ/r37y/d5vPPToDbYUbBoCIjqW+04OBg7NmzBxEREahcuTL+/PNPWFhY4Ny5cxg/fjwSExORN29emJiYYMuWLbh+/TocHBy0XT5lAJ+HlEOHDmHQoEHw9fWFhYUF/vnnH0ycOBGmpqa4dOkSgE9jVjw8PODl5YV+/fpps3TKgMLDw5E3b16kpKRgx44daNiwobTsa2NTvjaWhX5/3PUjIwqFAjt37kSTJk3w4cMHfPjwAX5+fujTpw8iIiLg6uqKefPmoVq1aggJCcGjR49w9OhRhhTSmNQP9j179mDz5s2oXbs2KleujGLFiqF169bYsGEDYmNj0adPHwBA9+7dsXv3buk6kaYkJibi1atXMDExga6uLtasWSNNjQcAa2tr6f+pR0H+PJgwpGQc7FGRkUuXLqFt27YYNWoUevTogbCwMJQuXRrGxsYoWbIkNmzYACsrK+ncKV9OASXShIiICDRu3BjXr19HjRo1sG/fPpXlY8aMwenTp/H333/D1NRUamc3O/2sbw3gDg0NhYuLC2rUqIEFCxYgX758WqiOtIU9Kloyc+ZMjB07VvolAHw6RLmrqyt69OiB0NBQ1KpVC82bN8e4ceNw8eJF9O3bFxERETAyMgIAhhTSiM+3QQCwsrLC+vXrUadOHVy9ehVr165VWZ4/f368e/cO8fHxKu0MKfQzPg8px44dw8aNG3H9+nU8f/4cTk5OOH36NIKDgzFixAhpAHeLFi2wZMkSbZZNvwB7VLRkyZIlGDhwIGbMmIERI0ZIb9A7d+6gYMGCaNasmfSFoVQqUbJkSYSEhKBRo0bYsmULz5VCGvH5l8PDhw+hUChgYmICOzs7PH78GJ6enoiLi8Off/4JDw8PvH79Gu7u7jAyMsK+ffvYvU4aN2zYMKxfvx56enrIkiUL7OzssHDhQpQtWxY3b95EjRo14OTkhKSkJHz8+BHXr1+XTsxKGZSgX06pVAohhFi5cqXQ0dERU6dOFcnJydLyp0+fisKFC4t9+/YJIYSIiIgQ7dq1E0uWLBHPnj3TSs2U8aRuh0IIMXHiRFG8eHFRqFAhkSNHDuHr6yuEECIkJEQ0bNhQGBkZiYIFC4oWLVqIevXqifj4eCGEECkpKVqpnTKOz7fDoKAgUaJECXHy5EkREREhdu/eLVq0aCGcnZ3FlStXhBBCPHjwQEyZMkVMnz5d+tz8/POTMh4GlV9MqVRKb0ylUin++usvoaOjI6ZNmyZ96IeHh4uSJUsKDw8PERoaKsaMGSPKlSsnXr9+rc3SKYOaMmWKsLGxEYGBgSI2Nla0aNFCWFhYiNu3bwshhHj06JFo1KiRKFmypFi4cKF0u4SEBC1VTBnR+vXrRb9+/USvXr1U2i9evCjq168v3N3dRWxsrBBCNdwwpGR83H+gBQqFAocPH8bQoUNRpkwZ6Rwqs2bNghAClpaW6NChA44fPw5XV1ds2LABPj4+sLW11XbplAF8PiZFqVTiwoULWLhwIerWrYugoCAcO3YMM2bMQJEiRZCcnIw8efJg/vz5yJ49O/bv34+AgAAAgKGhobaeAmUA4otRB7t27cKyZctw7do1JCYmSu1ly5ZFlSpVcOrUKaSkpABQndHDc0hlAtpOSpnRjh07hLGxsZg6daq4ePGiEEIIX19faTeQEEIkJiaK27dvi6CgIPH06VNtlksZ1IQJE8SsWbNEzpw5xb1790RwcLDIkiWL8Pb2FkII8eHDBzF27FgRGhoqhBDi/v37onHjxqJs2bIiICBAm6XTb+7zHhF/f3+xYcMGIYQQ/fr1ExYWFmLZsmUiKipKWicwMFAUKlRI2hYpc2FQ+cXu3bsn8uTJI5YvX55m2YoVK6TdQESa9vl4ks2bNwtHR0dx69Yt0bFjR1GvXj1hYmIiVq9eLa3z/PlzUaVKFbFhwwbptnfu3BGtW7cWYWFhv7x+yhg+3w5v3bolSpUqJUqUKCF2794thBDC3d1d5M+fX0yfPl2EhISIkJAQUatWLVGtWjWVgEOZB/vMfrEnT55AX19f5QiLqTMvevXqBVNTU3Tq1AmGhoYYNmyYFiuljCZ1ds/x48dx7NgxDB06FEWLFpUOJFirVi1069YNwKeTYfbo0QO6urpo3749dHR0oFQqUahQIWzcuJGzLOiHpW6Hw4cPx+PHj2FsbIy7d+9i8ODB+PjxI9atW4du3bph3LhxWLJkCSpVqoQsWbJgy5YtUCgU3zzWCmVcDCq/WGxsrMrxJ5RKpbS/9dixYyhTpgy2bNmict4UIk159eoVunfvjvDwcIwZMwYA0Lt3bzx8+BBHjx5FqVKlkD9/fjx58gQJCQm4ePEidHV1VQ7mxjEB9LPWrVuHVatW4ciRI8iTJw8SExPh7u6OmTNnQkdHB2vWrIGJiQm2bt2K+vXro23btjA0NERSUhIMDAy0XT79Yoylv1iJEiXw9u1b+Pr6Avj06yI1qOzevRsbN25Ey5YtUbhwYW2WSRmUnZ0dAgICkD17duzduxeXL1+Grq4u5s6diylTpqBmzZqws7NDmzZtvnn2WR47hX5WSEgIihUrhpIlS8Lc3Bx2dnZYs2YNdHV1MXjwYOzcuRNLly5F7dq1sWDBAuzZswcxMTEMKZkUfxr9Ynny5MHSpUvRu3dvJCcno3PnztDV1cW6deuwbt06nD17lkf4pHTl4uKCHTt2wN3dHT4+Pujfvz9cXFzQtGlTNG3aVGXdlJQU9qCQxoj/P1GgoaEhEhISkJSUBCMjIyQnJyNnzpyYOXMmGjduDC8vLxgbG2Pjxo1o3749hg0bBj09Pbi5uWn7KZAW8Mi0WqBUKrFjxw54eHjA1NQURkZG0NXVxaZNm1CqVCltl0eZxNWrV9GjRw+UKVMGAwcORNGiRbVdEmUSN2/eRKlSpTB+/HhMnDhRag8MDMTKlSvx/v17pKSk4NixYwCArl27Yvz48cibN6+WKiZtYlDRohcvXiAsLAwKhQJ58uRB9uzZtV0SZTJXr16Fh4cHcufOjTlz5iBPnjzaLokyiXXr1qFXr14YNGgQ2rRpA0tLSwwYMAAVK1ZEixYtULRoUezfvx8NGjTQdqmkZQwqRJnchQsX4OPjg1WrVnE2Bf1SO3bsQN++fWFgYAAhBGxtbXHmzBm8fv0aderUwfbt2+Hi4qLtMknLGFSISBo7wKmf9Ks9f/4cT58+RXJyMipVqgQdHR2MHj0au3btQnBwMOzs7LRdImkZgwoRAfhfWCHSltu3b2P27Nn4+++/cfjwYZQsWVLbJZEMcDg/EQHgtGPSro8fPyIpKQm2trY4fvw4B3eThD0qREQkG8nJyTzyMalgUCEiIiLZ4qg5IiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSL6rRw7dgwKhQKRkZHffRsnJyd4eXmlW01ElH4YVIhIo7p06QKFQoHevXunWebp6QmFQoEuXbr8+sKI6LfEoEJEGufo6IjNmzcjPj5eaktISMDGjRuRK1cuLVZGRL8bBhUi0rjSpUvD0dERAQEBUltAQABy5cqFUqVKSW2JiYkYMGAAbG1tYWRkhMqVK+PixYsq9/X333+jQIECMDY2Ro0aNRAaGprm8U6dOoUqVarA2NgYjo6OGDBgAOLi4tLt+RHRr8OgQkTpolu3bli7dq10fc2aNejatavKOiNGjMCOHTuwfv16XLlyBc7OzqhXrx4iIiIAAE+fPkXLli3RpEkTXLt2DT169MCoUaNU7uPhw4eoX78+WrVqhRs3bmDLli04deoU+vXrl/5PkojSHYMKEaWLjh074tSpUwgLC0NYWBhOnz6Njh07Ssvj4uLg7e2NuXPnokGDBihSpAhWrlwJY2NjrF69GgDg7e2NfPnyYf78+ShYsCA6dOiQZnzLzJkz0aFDBwwaNAj58+dHxYoVsXjxYmzYsAEJCQm/8ikTUTrgSQmJKF3Y2NigUaNGWLduHYQQaNSoEaytraXlDx8+RHJyMipVqiS16evr448//sCdO3cAAHfu3EH58uVV7rdChQoq169fv44bN27A399fahNCQKlU4vHjxyhcuHB6PD0i+kUYVIgo3XTr1k3aBbNs2bJ0eYzY2Fh4eHhgwIABaZZx4C7R749BhYjSTf369ZGUlASFQoF69eqpLMuXLx8MDAxw+vRp5M6dG8CnM+devHgRgwYNAgAULlwYe/bsUbnduXPnVK6XLl0a//zzD5ydndPviRCR1nCMChGlG11dXdy5cwf//PMPdHV1VZaZmpqiT58+GD58OA4ePIh//vkHPXv2xIcPH9C9e3cAQO/evfHgwQMMHz4c9+7dw8aNG7Fu3TqV+xk5ciTOnDmDfv364dq1a3jw4AF2797NwbREGQSDChGlKzMzM5iZmX112axZs9CqVSt06tQJpUuXRkhICAIDA2FpaQng066bHTt2YNeuXShRogR8fHwwY8YMlftwcXHB8ePHcf/+fVSpUgWlSpXChAkTYG9vn+7PjYjSn0IIIbRdBBEREdHXsEeFiIiIZItBhYiIiGSLQYWIiIhki0GFiIiIZItBhYiIiGSLQYWIiIhki0GFiIiIZItBhYiIiGSLQYWIiIhki0GFiIiIZItBhYiIiGSLQYWIiIhk6/8AHoK08GWUizwAAAAASUVORK5CYII=",
+ "text/plain": [
+ "
"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "## calculate avg response time\n",
+ "unique_models = set(result[\"response\"]['model'] for result in result[\"results\"])\n",
+ "model_dict = {model: {\"response_time\": []} for model in unique_models}\n",
+ "for completion_result in result[\"results\"]:\n",
+ " model_dict[completion_result[\"response\"][\"model\"]][\"response_time\"].append(completion_result[\"response_time\"])\n",
+ "\n",
+ "avg_response_time = {}\n",
+ "for model, data in model_dict.items():\n",
+ " avg_response_time[model] = sum(data[\"response_time\"]) / len(data[\"response_time\"])\n",
+ "\n",
+ "models = list(avg_response_time.keys())\n",
+ "response_times = list(avg_response_time.values())\n",
+ "\n",
+ "plt.bar(models, response_times)\n",
+ "plt.xlabel('Model', fontsize=10)\n",
+ "plt.ylabel('Average Response Time')\n",
+ "plt.title('Average Response Times for each Model')\n",
+ "\n",
+ "plt.xticks(models, [model[:15]+'...' if len(model) > 15 else model for model in models], rotation=45)\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "inSDIE3_IRds"
+ },
+ "source": [
+ "# Duration Test endpoint\n",
+ "\n",
+ "Run load testing for 2 mins. Hitting endpoints with 100+ queries every 15 seconds."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {
+ "id": "ePIqDx2EIURH"
+ },
+ "outputs": [],
+ "source": [
+ "models=[\"gpt-3.5-turbo\", \"replicate/llama-2-70b-chat:58d078176e02c219e11eb4da5a02a7830a283b14cf8f94537af893ccff5ee781\", \"claude-instant-1\"]\n",
+ "context = \"\"\"Paul Graham (/ɔrƦm/; born 1964)[3] is an English computer scientist, essayist, entrepreneur, venture capitalist, and author. He is best known for his work on the programming language Lisp, his former startup Viaweb (later renamed Yahoo! Store), cofounding the influential startup accelerator and seed capital firm Y Combinator, his essays, and Hacker News. He is the author of several computer programming books, including: On Lisp,[4] ANSI Common Lisp,[5] and Hackers & Painters.[6] Technology journalist Steven Levy has described Graham as a \"hacker philosopher\".[7] Graham was born in England, where he and his family maintain permanent residence. However he is also a citizen of the United States, where he was educated, lived, and worked until 2016.\"\"\"\n",
+ "prompt = \"Where does Paul Graham live?\"\n",
+ "final_prompt = context + prompt\n",
+ "result = load_test_model(models=models, prompt=final_prompt, num_calls=100, interval=15, duration=120)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 552
+ },
+ "id": "k6rJoELM6t1K",
+ "outputId": "f4968b59-3bca-4f78-a88b-149ad55e3cf7"
+ },
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAIXCAYAAABghH+YAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABwdUlEQVR4nO3dd1QU198G8GfpoNKUooKCYuwIaiL2GrGLJnYFOxrsNZbYFTsYG2JDjV2xRKOIir33EhsWLBGwUaXJ3vcPX+bnCiYsLi6Oz+ecPbp37ux+lx3YZ+/cmVEIIQSIiIiIZEJH2wUQERERaRLDDREREckKww0RERHJCsMNERERyQrDDREREckKww0RERHJCsMNERERyQrDDREREckKww0RERHJCsMNEcmSg4MDunfvru0y1DZnzhyUKFECurq6cHFx0XY5GnfkyBEoFAps27ZN26WoTaFQYNKkSWqv9+jRIygUCgQFBWm8Jsoaww19tiVLlkChUKBatWraLiXPcXBwgEKhkG758uXDDz/8gLVr12q7tK9Oxodidm5fqwMHDmDUqFGoWbMmVq9ejRkzZmi7pDwnKChIep9PnDiRabkQAvb29lAoFGjRooUWKqS8QE/bBdDXb/369XBwcMC5c+cQHh4OJycnbZeUp7i4uGD48OEAgOfPn2PFihXw8vJCSkoK+vTpo+Xqvh5ly5bFunXrVNrGjBmD/PnzY9y4cZn637lzBzo6X9f3t8OHD0NHRwcrV66EgYGBtsvJ04yMjLBhwwbUqlVLpf3o0aN4+vQpDA0NtVQZ5QUMN/RZHj58iFOnTiE4OBje3t5Yv349Jk6c+EVrUCqVSE1NhZGR0Rd93uwqWrQounbtKt3v3r07SpQoAT8/P4YbNdjY2Kj8HAFg5syZKFSoUKZ2AF/lh1t0dDSMjY01FmyEEEhOToaxsbFGHi8vadasGbZu3Yrff/8denr/+yjbsGEDqlSpgpcvX2qxOtK2r+trDeU569evh4WFBZo3b46ff/4Z69evl5alpaXB0tISPXr0yLReXFwcjIyMMGLECKktJSUFEydOhJOTEwwNDWFvb49Ro0YhJSVFZV2FQoEBAwZg/fr1KF++PAwNDbF//34AwNy5c1GjRg0ULFgQxsbGqFKlSpb79pOSkjBo0CAUKlQIBQoUQKtWrfDs2bMs96k/e/YMPXv2hI2NDQwNDVG+fHmsWrUqxz8zKysrlClTBvfv31dpVyqV8Pf3R/ny5WFkZAQbGxt4e3vjzZs3Kv0uXLgAd3d3FCpUCMbGxnB0dETPnj2l5Rn79+fOnQs/Pz8UL14cxsbGqFu3Lm7cuJGpnsOHD6N27drIly8fzM3N0bp1a9y6dUulz6RJk6BQKBAeHo7u3bvD3NwcZmZm6NGjB96+favSNzQ0FLVq1YK5uTny58+P0qVLY+zYsSp9svtef46P59xk7M44ceIEBg0aBCsrK5ibm8Pb2xupqamIiYmBp6cnLCwsYGFhgVGjRkEIofKYmnqPsqJQKLB69WokJiZKu10y5mi8e/cOU6dORcmSJWFoaAgHBweMHTs208/LwcEBLVq0QEhICKpWrQpjY2MsW7bsX5/37NmzaNKkCczMzGBiYoK6devi5MmTKn0iIiLwyy+/oHTp0jA2NkbBggXRrl07PHr0KNPjxcTEYOjQoXBwcIChoSHs7Ozg6emZKWwolUpMnz4ddnZ2MDIyQsOGDREeHv6vtX6oU6dOePXqFUJDQ6W21NRUbNu2DZ07d85yncTERAwfPhz29vYwNDRE6dKlMXfu3Ezvc0pKCoYOHQorKyvp78PTp0+zfExN/30gDRFEn6FMmTKiV69eQgghjh07JgCIc+fOSct79uwpzM3NRUpKisp6a9asEQDE+fPnhRBCpKeni8aNGwsTExMxZMgQsWzZMjFgwAChp6cnWrdurbIuAFG2bFlhZWUlJk+eLBYvXiwuX74shBDCzs5O/PLLL2LRokVi/vz54ocffhAAxJ49e1Qeo3379gKA6Natm1i8eLFo3769qFSpkgAgJk6cKPWLjIwUdnZ2wt7eXkyZMkUsXbpUtGrVSgAQfn5+//nzKV68uGjevLlKW1pamrC1tRU2NjYq7b179xZ6enqiT58+IiAgQIwePVrky5dPfP/99yI1NVUIIURUVJSwsLAQ3333nZgzZ45Yvny5GDdunChbtqz0OA8fPhQARMWKFYWDg4OYNWuWmDx5srC0tBRWVlYiMjJS6hsaGir09PTEd999J2bPni0mT54sChUqJCwsLMTDhw+lfhMnThQAhKurq2jbtq1YsmSJ6N27twAgRo0aJfW7ceOGMDAwEFWrVhULFiwQAQEBYsSIEaJOnTpSH3Xe6/9Svnx5Ubdu3U/+7L28vKT7q1evFgCEi4uLaNKkiVi8eLHo1q2b9Bpq1aolOnfuLJYsWSJatGghAIg1a9bkynuUlXXr1onatWsLQ0NDsW7dOrFu3Tpx//59IYQQXl5eAoD4+eefxeLFi4Wnp6cAIDw8PDK9ZicnJ2FhYSF+/fVXERAQIMLCwj75nIcOHRIGBgaievXqYt68ecLPz084OzsLAwMDcfbsWanf1q1bRaVKlcSECRNEYGCgGDt2rLCwsBDFixcXiYmJUr/4+HhRoUIFoaurK/r06SOWLl0qpk6dKr7//nvpdzQsLEzalqpUqSL8/PzEpEmThImJifjhhx/+9Wf04ft4/vx5UaNGDdGtWzdp2c6dO4WOjo549uxZpt89pVIpGjRoIBQKhejdu7dYtGiRaNmypQAghgwZovIcXbt2FQBE586dxaJFi0Tbtm2Fs7Nzjv8+ZPxOrl69+j9fH2kGww3l2IULFwQAERoaKoR4/8fDzs5ODB48WOoTEhIiAIg///xTZd1mzZqJEiVKSPfXrVsndHR0xPHjx1X6BQQECADi5MmTUhsAoaOjI27evJmpprdv36rcT01NFRUqVBANGjSQ2i5evJjlH7Tu3btn+uPVq1cvUbhwYfHy5UuVvh07dhRmZmaZnu9jxYsXF40bNxYvXrwQL168ENevX5c+UH18fKR+x48fFwDE+vXrVdbfv3+/SvuOHTtUQmFWMv6QGhsbi6dPn0rtZ8+eFQDE0KFDpTYXFxdhbW0tXr16JbVdvXpV6OjoCE9PT6ktI9z07NlT5bnatGkjChYsKN338/MTAMSLFy8+WZ867/V/yUm4cXd3F0qlUmqvXr26UCgUol+/flLbu3fvhJ2dncpja/I9+hQvLy+RL18+lbYrV64IAKJ3794q7SNGjBAAxOHDh1VeMwCxf//+/3wupVIpSpUqlenn8fbtW+Ho6Ch+/PFHlbaPnT59WgAQa9euldomTJggAIjg4OAsn0+I/4WbsmXLqnzpWbBggQAgrl+//q91fxhuFi1aJAoUKCDV165dO1G/fn3pZ/FhuNm5c6cAIKZNm6byeD///LNQKBQiPDxcCPG/n/cvv/yi0q9z5845/vvAcPPlcbcU5dj69ethY2OD+vXrA3g/rN6hQwds2rQJ6enpAIAGDRqgUKFC2Lx5s7TemzdvEBoaig4dOkhtW7duRdmyZVGmTBm8fPlSujVo0AAAEBYWpvLcdevWRbly5TLV9OHcgjdv3iA2Nha1a9fGpUuXpPaMXVi//PKLyroDBw5UuS+EwPbt29GyZUsIIVTqcnd3R2xsrMrjfsqBAwdgZWUFKysrVKxYEevWrUOPHj0wZ84clddvZmaGH3/8UeV5qlSpgvz580uv39zcHACwZ88epKWl/evzenh4oGjRotL9H374AdWqVcNff/0F4P3k5itXrqB79+6wtLSU+jk7O+PHH3+U+n2oX79+Kvdr166NV69eIS4uTqW+Xbt2QalUZlmXuu+1pvXq1UvliKpq1apBCIFevXpJbbq6uqhatSoePHigUrem36PsyHgfhg0bptKeMUl97969Ku2Ojo5wd3f/z8e9cuUK7t27h86dO+PVq1fS60lMTETDhg1x7Ngx6T388PcqLS0Nr169gpOTE8zNzVV+B7Zv345KlSqhTZs2mZ7v46PYevTooTK3qHbt2gCg8jP/L+3bt0dSUhL27NmD+Ph47Nmz55O7pP766y/o6upi0KBBKu3Dhw+HEAL79u2T+gHI1G/IkCEq9zX194Fyxzcdbo4dO4aWLVuiSJEiUCgU2LlzZ64/57Nnz9C1a1dpTkjFihVx4cKFXH9eTUtPT8emTZtQv359PHz4EOHh4QgPD0e1atUQFRWFQ4cOAQD09PTw008/YdeuXdL8gODgYKSlpamEm3v37uHmzZtSCMi4fffddwDeT7T8kKOjY5Z17dmzB25ubjAyMoKlpSWsrKywdOlSxMbGSn0iIiKgo6OT6TE+PsrrxYsXiImJQWBgYKa6MuYRfVxXVqpVq4bQ0FDs378fc+fOhbm5Od68eaPyh/3evXuIjY2FtbV1pudKSEiQnqdu3br46aefMHnyZBQqVAitW7fG6tWrs5yrUqpUqUxt3333nTRPIiIiAgBQunTpTP3Kli0rfdB9qFixYir3LSwsAECac9KhQwfUrFkTvXv3ho2NDTp27IgtW7aoBB1132tN+/g1mJmZAQDs7e0ztX84lyY33qPsyNheP94+bW1tYW5uLr2PGT71u/Gxe/fuAQC8vLwyvZ4VK1YgJSVF+r1JSkrChAkTpLkqhQoVgpWVFWJiYlR+t+7fv48KFSpk6/n/a1vKDisrKzRq1AgbNmxAcHAw0tPT8fPPP2fZNyIiAkWKFEGBAgVU2suWLSstz/hXR0cHJUuWVOn38e+Jpv4+UO74po+WSkxMRKVKldCzZ0+0bds215/vzZs3qFmzJurXr499+/bBysoK9+7dk36pvyaHDx/G8+fPsWnTJmzatCnT8vXr16Nx48YAgI4dO2LZsmXYt28fPDw8sGXLFpQpUwaVKlWS+iuVSlSsWBHz58/P8vk+/uDJ6uiP48ePo1WrVqhTpw6WLFmCwoULQ19fH6tXr8aGDRvUfo0ZH8hdu3aFl5dXln2cnZ3/83EKFSqERo0aAQDc3d1RpkwZtGjRAgsWLJC+jSuVSlhbW6tMyP6QlZUVAEgnPztz5gz+/PNPhISEoGfPnpg3bx7OnDmD/Pnzq/061aGrq5tlu/j/CZnGxsY4duwYwsLCsHfvXuzfvx+bN29GgwYNcODAAejq6qr9Xmvap15DVu3ig4mm2n6Psnv+nuweGZWxfc+ZM+eTJwvMqHXgwIFYvXo1hgwZgurVq8PMzAwKhQIdO3b85Ajdf/mvbSm7OnfujD59+iAyMhJNmzaVRs5ym6b+PlDu+KbDTdOmTdG0adNPLk9JScG4ceOwceNGxMTEoEKFCpg1axbq1auXo+ebNWsW7O3tsXr1aqktu9+y8pr169fD2toaixcvzrQsODgYO3bsQEBAAIyNjVGnTh0ULlwYmzdvRq1atXD48OFM5yUpWbIkrl69ioYNG+b4JGzbt2+HkZERQkJCVA4D/vDnDQDFixeHUqnEw4cPVUY3Pj5SI+NIifT0dCmcaELz5s1Rt25dzJgxA97e3siXLx9KliyJgwcPombNmtn6cHJzc4ObmxumT5+ODRs2oEuXLti0aRN69+4t9cn4Zv6hu3fvwsHBAcD7nwPw/nwwH7t9+zYKFSqEfPnyqf36dHR00LBhQzRs2BDz58/HjBkzMG7cOISFhaFRo0Yaea+1ITfeo+zI2F7v3bsnjTIAQFRUFGJiYqT3UV0ZIxOmpqb/uX1v27YNXl5emDdvntSWnJyMmJiYTI+Z1RF5ualNmzbw9vbGmTNnVHZ/f6x48eI4ePAg4uPjVUZvbt++LS3P+FepVOL+/fsqozUf/57k1t8H0oxverfUfxkwYABOnz6NTZs24dq1a2jXrh2aNGmS5YdGduzevRtVq1ZFu3btYG1tDVdXVyxfvlzDVee+pKQkBAcHo0WLFvj5558z3QYMGID4+Hjs3r0bwPsPu59//hl//vkn1q1bh3fv3qnskgLe7zt/9uxZlj+PpKSkTLtHsqKrqwuFQiHN9wHeHxb98e7GjPkIS5YsUWlfuHBhpsf76aefsH379iz/YL948eI/a/qU0aNH49WrV9Lrbd++PdLT0zF16tRMfd+9eyd9iLx58ybTN9uMb90f7/bYuXMnnj17Jt0/d+4czp49KwX6woULw8XFBWvWrFH5kLpx4wYOHDiAZs2aqf26Xr9+nant4/o08V5rQ268R9mR8T74+/urtGeMfDVv3lztxwSAKlWqoGTJkpg7dy4SEhIyLf9w+9bV1c30mhYuXKjyuwYAP/30E65evYodO3Zkejx1R2SyK3/+/Fi6dCkmTZqEli1bfrJfs2bNkJ6ejkWLFqm0+/n5QaFQSL8XGf/+/vvvKv0+/vnn5t8H+nzf9MjNv3n8+DFWr16Nx48fo0iRIgCAESNGYP/+/Tk+LfqDBw+wdOlSDBs2DGPHjsX58+cxaNAgGBgYfHJYMy/avXs34uPj0apVqyyXu7m5wcrKCuvXr5dCTIcOHbBw4UJMnDgRFStWVPkGCgDdunXDli1b0K9fP4SFhaFmzZpIT0/H7du3sWXLFum8Hf+mefPmmD9/Ppo0aYLOnTsjOjoaixcvhpOTE65duyb1q1KlCn766Sf4+/vj1atXcHNzw9GjR3H37l0AqsP/M2fORFhYGKpVq4Y+ffqgXLlyeP36NS5duoSDBw9m+WGeHU2bNkWFChUwf/58+Pj4oG7duvD29oavry+uXLmCxo0bQ19fH/fu3cPWrVuxYMEC/Pzzz1izZg2WLFmCNm3aoGTJkoiPj8fy5cthamqaKYw4OTmhVq1a6N+/P1JSUuDv74+CBQti1KhRUp85c+agadOmqF69Onr16oWkpCQsXLgQZmZmObqGzpQpU3Ds2DE0b94cxYsXR3R0NJYsWQI7OzvpTLKaeK+1ITfeo+yoVKkSvLy8EBgYiJiYGNStWxfnzp3DmjVr4OHhIU3oV5eOjg5WrFiBpk2bonz58ujRoweKFi2KZ8+eISwsDKampvjzzz8BAC1atMC6detgZmaGcuXK4fTp0zh48CAKFiyo8pgjR47Etm3b0K5dO/Ts2RNVqlTB69evsXv3bgQEBKjsitak7Pz9bNmyJerXr49x48bh0aNHqFSpEg4cOIBdu3ZhyJAh0kiWi4sLOnXqhCVLliA2NhY1atTAoUOHsjwHT279fSAN0MoxWnkQALFjxw7p/p49ewQAkS9fPpWbnp6eaN++vRBCiFu3bgkA/3obPXq09Jj6+vqievXqKs87cOBA4ebm9kVeo6a0bNlSGBkZqZzf4mPdu3cX+vr60iGSSqVS2NvbZ3koZobU1FQxa9YsUb58eWFoaCgsLCxElSpVxOTJk0VsbKzUDx8dRv2hlStXilKlSglDQ0NRpkwZsXr1aukw5g8lJiYKHx8fYWlpKfLnzy88PDzEnTt3BAAxc+ZMlb5RUVHCx8dH2NvbC319fWFraysaNmwoAgMD//NnldV5bjIEBQVlOjw0MDBQVKlSRRgbG4sCBQqIihUrilGjRol//vlHCCHEpUuXRKdOnUSxYsWEoaGhsLa2Fi1atBAXLlyQHiPjsNM5c+aIefPmCXt7e2FoaChq164trl69mqmOgwcPipo1awpjY2NhamoqWrZsKf7++2+VPhk/w48P8c44LDfjnDiHDh0SrVu3FkWKFBEGBgaiSJEiolOnTuLu3bsq62X3vf4vOTkU/ONDtD/12rI6LFsIzbxHn/Kp50xLSxOTJ08Wjo6OQl9fX9jb24sxY8aI5OTkTK/5U9vbp1y+fFm0bdtWFCxYUBgaGorixYuL9u3bi0OHDkl93rx5I3r06CEKFSok8ufPL9zd3cXt27cz/YyFEOLVq1diwIABomjRosLAwEDY2dkJLy8v6W9BxqHgW7duVVkvu4dLf+p9/FhWP4v4+HgxdOhQUaRIEaGvry9KlSol5syZo3IovBBCJCUliUGDBomCBQuKfPnyiZYtW4onT55kOhRciOz9feCh4F+eQohcGiv8yigUCuzYsQMeHh4AgM2bN6NLly64efNmpolv+fPnh62tLVJTU//zsMWCBQtKEw2LFy+OH3/8EStWrJCWL126FNOmTVPZfUDaceXKFbi6uuKPP/5Aly5dtF1Ojj169AiOjo6YM2eOyhmgiYi+Fdwt9Qmurq5IT09HdHS0dP6FjxkYGKBMmTLZfsyaNWtmmpR29+7dHE8IpJxLSkrKNCnU398fOjo6qFOnjpaqIiIiTfimw01CQoLKftSHDx/iypUrsLS0xHfffYcuXbrA09MT8+bNg6urK168eIFDhw7B2dk5R5P4hg4diho1amDGjBlo3749zp07h8DAQAQGBmryZVE2zJ49GxcvXkT9+vWhp6eHffv2Yd++fejbt2+uH4pMRES5TNv7xbQpY9/vx7eMfcipqaliwoQJwsHBQejr64vChQuLNm3aiGvXruX4Of/8809RoUIFaU5IduZtkOYdOHBA1KxZU1hYWAh9fX1RsmRJMWnSJJGWlqbt0j7bh3NuiIi+RZxzQ0RERLLC89wQERGRrDDcEBERkax8cxOKlUol/vnnHxQoUOCrOvU7ERHRt0wIgfj4eBQpUgQ6Ov8+NvPNhZt//vmHR8MQERF9pZ48eQI7O7t/7fPNhZuMC6Y9efIEpqamWq6GiIiIsiMuLg729vYqFz79lG8u3GTsijI1NWW4ISIi+spkZ0oJJxQTERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGs6Gm7ACLSLIdf92q7BNKyRzOba/X5uQ2StrdBjtwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrGg13CxduhTOzs4wNTWFqakpqlevjn379v3rOlu3bkWZMmVgZGSEihUr4q+//vpC1RIREdHXQKvhxs7ODjNnzsTFixdx4cIFNGjQAK1bt8bNmzez7H/q1Cl06tQJvXr1wuXLl+Hh4QEPDw/cuHHjC1dOREREeZVCCCG0XcSHLC0tMWfOHPTq1SvTsg4dOiAxMRF79uyR2tzc3ODi4oKAgIBsPX5cXBzMzMwQGxsLU1NTjdVNlFfwooWk7YsWchuk3NgG1fn8zjNzbtLT07Fp0yYkJiaievXqWfY5ffo0GjVqpNLm7u6O06dPf/JxU1JSEBcXp3IjIiIi+dJ6uLl+/Try588PQ0ND9OvXDzt27EC5cuWy7BsZGQkbGxuVNhsbG0RGRn7y8X19fWFmZibd7O3tNVo/ERER5S1aDzelS5fGlStXcPbsWfTv3x9eXl74+++/Nfb4Y8aMQWxsrHR78uSJxh6biIiI8h49bRdgYGAAJycnAECVKlVw/vx5LFiwAMuWLcvU19bWFlFRUSptUVFRsLW1/eTjGxoawtDQULNFExERUZ6l9ZGbjymVSqSkpGS5rHr16jh06JBKW2ho6Cfn6BAREdG3R6sjN2PGjEHTpk1RrFgxxMfHY8OGDThy5AhCQkIAAJ6enihatCh8fX0BAIMHD0bdunUxb948NG/eHJs2bcKFCxcQGBiozZdBREREeYhWw010dDQ8PT3x/PlzmJmZwdnZGSEhIfjxxx8BAI8fP4aOzv8Gl2rUqIENGzZg/PjxGDt2LEqVKoWdO3eiQoUK2noJRERElMdoNdysXLnyX5cfOXIkU1u7du3Qrl27XKqIiIiIvnZ5bs4NERER0edguCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWdFquPH19cX333+PAgUKwNraGh4eHrhz586/rhMUFASFQqFyMzIy+kIVExERUV6n1XBz9OhR+Pj44MyZMwgNDUVaWhoaN26MxMTEf13P1NQUz58/l24RERFfqGIiIiLK6/S0+eT79+9XuR8UFARra2tcvHgRderU+eR6CoUCtra2uV0eERERfYXy1Jyb2NhYAIClpeW/9ktISEDx4sVhb2+P1q1b4+bNm5/sm5KSgri4OJUbERERyVeeCTdKpRJDhgxBzZo1UaFChU/2K126NFatWoVdu3bhjz/+gFKpRI0aNfD06dMs+/v6+sLMzEy62dvb59ZLICIiojwgz4QbHx8f3LhxA5s2bfrXftWrV4enpydcXFxQt25dBAcHw8rKCsuWLcuy/5gxYxAbGyvdnjx5khvlExERUR6h1Tk3GQYMGIA9e/bg2LFjsLOzU2tdfX19uLq6Ijw8PMvlhoaGMDQ01ESZRERE9BXQ6siNEAIDBgzAjh07cPjwYTg6Oqr9GOnp6bh+/ToKFy6cCxUSERHR10arIzc+Pj7YsGEDdu3ahQIFCiAyMhIAYGZmBmNjYwCAp6cnihYtCl9fXwDAlClT4ObmBicnJ8TExGDOnDmIiIhA7969tfY6iIiIKO/QarhZunQpAKBevXoq7atXr0b37t0BAI8fP4aOzv8GmN68eYM+ffogMjISFhYWqFKlCk6dOoVy5cp9qbKJiIgoD9NquBFC/GefI0eOqNz38/ODn59fLlVEREREX7s8MaFYThx+3avtEkjLHs1sru0SiIi+aXnmUHAiIiIiTWC4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZyVG4uX//PsaPH49OnTohOjoaALBv3z7cvHlTo8URERERqUvtcHP06FFUrFgRZ8+eRXBwMBISEgAAV69excSJEzVeIBEREZE61A43v/76K6ZNm4bQ0FAYGBhI7Q0aNMCZM2c0WhwRERGRutQON9evX0ebNm0ytVtbW+Ply5caKYqIiIgop9QON+bm5nj+/Hmm9suXL6No0aIaKYqIiIgop9QONx07dsTo0aMRGRkJhUIBpVKJkydPYsSIEfD09MyNGomIiIiyTe1wM2PGDJQpUwb29vZISEhAuXLlUKdOHdSoUQPjx4/PjRqJiIiIsk3tq4IbGBhg+fLl+O2333Djxg0kJCTA1dUVpUqVyo36iIiIiNSidrjJUKxYMRQrVkyTtRARERF9NrXDjRAC27ZtQ1hYGKKjo6FUKlWWBwcHa6w4IiIiInWpHW6GDBmCZcuWoX79+rCxsYFCociNuoiIiIhyRO1ws27dOgQHB6NZs2a5UQ8RERHRZ1H7aCkzMzOUKFEiN2ohIiIi+mxqh5tJkyZh8uTJSEpKyo16iIiIiD6L2rul2rdvj40bN8La2hoODg7Q19dXWX7p0iWNFUdERESkLrXDjZeXFy5evIiuXbtyQjERERHlOWqHm7179yIkJAS1atXKjXqIiIiIPovac27s7e1hamqaG7UQERERfTa1w828efMwatQoPHr0KBfKISIiIvo8au+W6tq1K96+fYuSJUvCxMQk04Ti169fa6w4IiIiInWpHW78/f1zoQwiIiIizcjR0VJEREREeVW2wk1cXJw0iTguLu5f+3KyMREREWlTtsKNhYUFnj9/Dmtra5ibm2d5bhshBBQKBdLT0zVeJBEREVF2ZSvcHD58GJaWlgCAsLCwXC2IiIiI6HNkK9zUrVsXJUqUwPnz51G3bt3cromIiIgox7J9nptHjx5xlxMRERHleWqfxI+IiIgoL1PrUPCQkBCYmZn9a59WrVp9VkFEREREn0OtcPNf57jh0VJERESkbWrtloqMjIRSqfzkjcGGiIiItC3b4Sarc9sQERER5TXZDjdCiNysg4iIiEgjsh1uvLy8YGxsnJu1EBEREX22bE8oXr16dW7WQURERKQRPM8NERERyQrDDREREckKww0RERHJSo7DTXh4OEJCQpCUlAQgZ0dT+fr64vvvv0eBAgVgbW0NDw8P3Llz5z/X27p1K8qUKQMjIyNUrFgRf/31l9rPTURERPKkdrh59eoVGjVqhO+++w7NmjXD8+fPAQC9evXC8OHD1Xqso0ePwsfHB2fOnEFoaCjS0tLQuHFjJCYmfnKdU6dOoVOnTujVqxcuX74MDw8PeHh44MaNG+q+FCIiIpIhtcPN0KFDoaenh8ePH8PExERq79ChA/bv36/WY+3fvx/du3dH+fLlUalSJQQFBeHx48e4ePHiJ9dZsGABmjRpgpEjR6Js2bKYOnUqKleujEWLFqn7UoiIiEiG1Lq2FAAcOHAAISEhsLOzU2kvVaoUIiIiPquY2NhYAIClpeUn+5w+fRrDhg1TaXN3d8fOnTuz7J+SkoKUlBTpflxc3GfVSERERHmb2iM3iYmJKiM2GV6/fg1DQ8McF6JUKjFkyBDUrFkTFSpU+GS/yMhI2NjYqLTZ2NggMjIyy/6+vr4wMzOTbvb29jmukYiIiPI+tcNN7dq1sXbtWum+QqGAUqnE7NmzUb9+/RwX4uPjgxs3bmDTpk05foysjBkzBrGxsdLtyZMnGn18IiIiylvU3i01e/ZsNGzYEBcuXEBqaipGjRqFmzdv4vXr1zh58mSOihgwYAD27NmDY8eOZdrd9TFbW1tERUWptEVFRcHW1jbL/oaGhp81okRERERfF7VHbipUqIC7d++iVq1aaN26NRITE9G2bVtcvnwZJUuWVOuxhBAYMGAAduzYgcOHD8PR0fE/16levToOHTqk0hYaGorq1aur9dxEREQkT2qP3ACAmZkZxo0b99lP7uPjgw0bNmDXrl0oUKCANG/GzMxMukinp6cnihYtCl9fXwDA4MGDUbduXcybNw/NmzfHpk2bcOHCBQQGBn52PURERPT1U3vkZv/+/Thx4oR0f/HixXBxcUHnzp3x5s0btR5r6dKliI2NRb169VC4cGHptnnzZqnP48ePpXPpAECNGjWwYcMGBAYGolKlSti2bRt27tz5r5OQiYiI6Nuh9sjNyJEjMWvWLADA9evXMWzYMAwfPhxhYWEYNmyYWlcPz85ZjY8cOZKprV27dmjXrl22n4eIiIi+HWqHm4cPH6JcuXIAgO3bt6Nly5aYMWMGLl26hGbNmmm8QCIiIiJ1qL1bysDAAG/fvgUAHDx4EI0bNwbw/sR7PEEeERERaZvaIze1atXCsGHDULNmTZw7d06aH3P37t3/PIybiIiIKLepPXKzaNEi6OnpYdu2bVi6dCmKFi0KANi3bx+aNGmi8QKJiIiI1KH2yE2xYsWwZ8+eTO1+fn4aKYiIiIjoc+ToPDdKpRLh4eGIjo6GUqlUWVanTh2NFEZERESUE2qHmzNnzqBz586IiIjIdCi3QqFAenq6xoojIiIiUpfa4aZfv36oWrUq9u7di8KFC0OhUORGXUREREQ5ona4uXfvHrZt2wYnJ6fcqIeIiIjos6h9tFS1atUQHh6eG7UQERERfTa1R24GDhyI4cOHIzIyEhUrVoS+vr7KcmdnZ40VR0RERKQutcPNTz/9BADo2bOn1KZQKCCE4IRiIiIi0rocXVuKiIiIKK9SO9wUL148N+ogIiIi0ogcncTv/v378Pf3x61btwAA5cqVw+DBg1GyZEmNFkdERESkLrWPlgoJCUG5cuVw7tw5ODs7w9nZGWfPnkX58uURGhqaGzUSERERZZvaIze//vorhg4dipkzZ2ZqHz16NH788UeNFUdERESkLrVHbm7duoVevXplau/Zsyf+/vtvjRRFRERElFNqhxsrKytcuXIlU/uVK1dgbW2tiZqIiIiIckzt3VJ9+vRB37598eDBA9SoUQMAcPLkScyaNQvDhg3TeIFERERE6lA73Pz2228oUKAA5s2bhzFjxgAAihQpgkmTJmHQoEEaL5CIiIhIHWqHG4VCgaFDh2Lo0KGIj48HABQoUEDjhRERERHlRI7OcwMA0dHRuHPnDgCgTJkysLKy0lhRRERERDml9oTi+Ph4dOvWDUWKFEHdunVRt25dFClSBF27dkVsbGxu1EhERESUbWqHm969e+Ps2bPYu3cvYmJiEBMTgz179uDChQvw9vbOjRqJiIiIsk3t3VJ79uxBSEgIatWqJbW5u7tj+fLlaNKkiUaLIyIiIlKX2iM3BQsWhJmZWaZ2MzMzWFhYaKQoIiIiopxSO9yMHz8ew4YNQ2RkpNQWGRmJkSNH4rffftNocURERETqUnu31NKlSxEeHo5ixYqhWLFiAIDHjx/D0NAQL168wLJly6S+ly5d0lylRERERNmgdrjx8PDIhTKIiIiINEPtcDNx4sTcqIOIiIhII9Sec/PkyRM8ffpUun/u3DkMGTIEgYGBGi2MiIiIKCfUDjedO3dGWFgYgPcTiRs1aoRz585h3LhxmDJlisYLJCIiIlKH2uHmxo0b+OGHHwAAW7ZsQcWKFXHq1CmsX78eQUFBmq6PiIiISC1qh5u0tDQYGhoCAA4ePIhWrVoBeH99qefPn2u2OiIiIiI1qR1uypcvj4CAABw/fhyhoaHSWYn/+ecfFCxYUOMFEhEREalD7XAza9YsLFu2DPXq1UOnTp1QqVIlAMDu3bul3VVERERE2qL2oeD16tXDy5cvERcXp3K5hb59+8LExESjxRERERGpS+2RGwAQQuDixYtYtmwZ4uPjAQAGBgYMN0RERKR1ao/cREREoEmTJnj8+DFSUlLw448/okCBApg1axZSUlIQEBCQG3USERERZYvaIzeDBw9G1apV8ebNGxgbG0vtbdq0waFDhzRaHBEREZG61B65OX78OE6dOgUDAwOVdgcHBzx79kxjhRERERHlhNojN0qlEunp6Znanz59igIFCmikKCIiIqKcUjvcNG7cGP7+/tJ9hUKBhIQETJw4Ec2aNdNkbURERERqU3u31Lx58+Du7o5y5cohOTkZnTt3xr1791CoUCFs3LgxN2okIiIiyja1R27s7Oxw9epVjBs3DkOHDoWrqytmzpyJy5cvw9raWq3HOnbsGFq2bIkiRYpAoVBg586d/9r/yJEjUCgUmW6RkZHqvgwiIiKSKbVHbgBAT08PXbp0QZcuXaS258+fY+TIkVi0aFG2HycxMRGVKlVCz5490bZt22yvd+fOHZiamkr31Q1VREREJF9qhZubN28iLCwMBgYGaN++PczNzfHy5UtMnz4dAQEBKFGihFpP3rRpUzRt2lStdYD3Ycbc3Fzt9YiIiEj+sr1bavfu3XB1dcWgQYPQr18/VK1aFWFhYShbtixu3bqFHTt24ObNm7lZq8TFxQWFCxfGjz/+iJMnT/5r35SUFMTFxanciIiISL6yHW6mTZsGHx8fxMXFYf78+Xjw4AEGDRqEv/76C/v375euDp6bChcujICAAGzfvh3bt2+Hvb096tWrh0uXLn1yHV9fX5iZmUk3e3v7XK+TiIiItCfb4ebOnTvw8fFB/vz5MXDgQOjo6MDPzw/ff/99btanonTp0vD29kaVKlVQo0YNrFq1CjVq1ICfn98n1xkzZgxiY2Ol25MnT75YvURERPTlZXvOTXx8vDSJV1dXF8bGxmrPsckNP/zwA06cOPHJ5YaGhjA0NPyCFREREZE2qTWhOCQkBGZmZgDen6n40KFDuHHjhkqfVq1aaa66bLhy5QoKFy78RZ+TiIiI8i61wo2Xl5fKfW9vb5X7CoUiy0szfEpCQgLCw8Ol+w8fPsSVK1dgaWmJYsWKYcyYMXj27BnWrl0LAPD394ejoyPKly+P5ORkrFixAocPH8aBAwfUeRlEREQkY9kON0qlUuNPfuHCBdSvX1+6P2zYMADvQ1RQUBCeP3+Ox48fS8tTU1MxfPhwPHv2DCYmJnB2dsbBgwdVHoOIiIi+bTk6iZ+m1KtXD0KITy4PCgpSuT9q1CiMGjUql6siIiKir5nal18gIiIiyssYboiIiEhWGG6IiIhIVhhuiIiISFZyFG5iYmKwYsUKjBkzBq9fvwYAXLp0Cc+ePdNocURERETqUvtoqWvXrqFRo0YwMzPDo0eP0KdPH1haWiI4OBiPHz+WzklDREREpA1qj9wMGzYM3bt3x71792BkZCS1N2vWDMeOHdNocURERETqUjvcnD9/PtOZiQGgaNGiiIyM1EhRRERERDmldrgxNDREXFxcpva7d+/CyspKI0URERER5ZTa4aZVq1aYMmUK0tLSALy/ntTjx48xevRo/PTTTxovkIiIiEgdaoebefPmISEhAdbW1khKSkLdunXh5OSEAgUKYPr06blRIxEREVG2qX20lJmZGUJDQ3HixAlcu3YNCQkJqFy5Mho1apQb9RERERGpJccXzqxVqxZq1aqlyVqIiIiIPpva4eb333/Psl2hUMDIyAhOTk6oU6cOdHV1P7s4IiIiInWpHW78/Pzw4sULvH37FhYWFgCAN2/ewMTEBPnz50d0dDRKlCiBsLAw2Nvba7xgIiIion+j9oTiGTNm4Pvvv8e9e/fw6tUrvHr1Cnfv3kW1atWwYMECPH78GLa2thg6dGhu1EtERET0r9QeuRk/fjy2b9+OkiVLSm1OTk6YO3cufvrpJzx48ACzZ8/mYeFERESkFWqP3Dx//hzv3r3L1P7u3TvpDMVFihRBfHz851dHREREpCa1w039+vXh7e2Ny5cvS22XL19G//790aBBAwDA9evX4ejoqLkqiYiIiLJJ7XCzcuVKWFpaokqVKjA0NIShoSGqVq0KS0tLrFy5EgCQP39+zJs3T+PFEhEREf0Xtefc2NraIjQ0FLdv38bdu3cBAKVLl0bp0qWlPvXr19dchURERERqyPFJ/MqUKYMyZcposhYiIiKiz5ajcPP06VPs3r0bjx8/Rmpqqsqy+fPna6QwIiIiopxQO9wcOnQIrVq1QokSJXD79m1UqFABjx49ghAClStXzo0aiYiIiLJN7QnFY8aMwYgRI3D9+nUYGRlh+/btePLkCerWrYt27drlRo1ERERE2aZ2uLl16xY8PT0BAHp6ekhKSkL+/PkxZcoUzJo1S+MFEhEREalD7XCTL18+aZ5N4cKFcf/+fWnZy5cvNVcZERERUQ6oPefGzc0NJ06cQNmyZdGsWTMMHz4c169fR3BwMNzc3HKjRiIiIqJsUzvczJ8/HwkJCQCAyZMnIyEhAZs3b0apUqV4pBQRERFpnVrhJj09HU+fPoWzszOA97uoAgICcqUwIiIiopxQa86Nrq4uGjdujDdv3uRWPURERESfRe0JxRUqVMCDBw9yoxYiIiKiz6Z2uJk2bRpGjBiBPXv24Pnz54iLi1O5EREREWmT2hOKmzVrBgBo1aoVFAqF1C6EgEKhQHp6uuaqIyIiIlKT2uEmLCwsN+ogIiIi0gi1w03dunVzow4iIiIijVB7zg0AHD9+HF27dkWNGjXw7NkzAMC6detw4sQJjRZHREREpC61w8327dvh7u4OY2NjXLp0CSkpKQCA2NhYzJgxQ+MFEhEREakjR0dLBQQEYPny5dDX15faa9asiUuXLmm0OCIiIiJ1qR1u7ty5gzp16mRqNzMzQ0xMjCZqIiIiIsoxtcONra0twsPDM7WfOHECJUqU0EhRRERERDmldrjp06cPBg8ejLNnz0KhUOCff/7B+vXrMWLECPTv3z83aiQiIiLKNrUPBf/111+hVCrRsGFDvH37FnXq1IGhoSFGjBiBgQMH5kaNRERERNmmdrhRKBQYN24cRo4cifDwcCQkJKBcuXLInz9/btRHREREpBa1d0v98ccfePv2LQwMDFCuXDn88MMPDDZERESUZ6gdboYOHQpra2t07twZf/3112ddS+rYsWNo2bIlihQpAoVCgZ07d/7nOkeOHEHlypVhaGgIJycnBAUF5fj5iYiISH7UDjfPnz/Hpk2boFAo0L59exQuXBg+Pj44deqU2k+emJiISpUqYfHixdnq//DhQzRv3hz169fHlStXMGTIEPTu3RshISFqPzcRERHJk9pzbvT09NCiRQu0aNECb9++xY4dO7BhwwbUr18fdnZ2uH//frYfq2nTpmjatGm2+wcEBMDR0RHz5s0DAJQtWxYnTpyAn58f3N3d1X0pREREJENqh5sPmZiYwN3dHW/evEFERARu3bqlqbqydPr0aTRq1Eilzd3dHUOGDPnkOikpKdIlIgAgLi4ut8ojIiKiPCBHF858+/Yt1q9fj2bNmqFo0aLw9/dHmzZtcPPmTU3XpyIyMhI2NjYqbTY2NoiLi0NSUlKW6/j6+sLMzEy62dvb52qNREREpF1qh5uOHTvC2toaQ4cORYkSJXDkyBGEh4dj6tSpKFOmTG7U+FnGjBmD2NhY6fbkyRNtl0RERES5SO3dUrq6utiyZQvc3d2hq6ursuzGjRuoUKGCxor7mK2tLaKiolTaoqKiYGpqCmNj4yzXMTQ0hKGhYa7VRERERHmL2uFm/fr1Kvfj4+OxceNGrFixAhcvXvysQ8P/S/Xq1fHXX3+ptIWGhqJ69eq59pxERET0dcnRnBvg/TlqvLy8ULhwYcydOxcNGjTAmTNn1HqMhIQEXLlyBVeuXAHw/lDvK1eu4PHjxwDe71Ly9PSU+vfr1w8PHjzAqFGjcPv2bSxZsgRbtmzB0KFDc/oyiIiISGbUGrmJjIxEUFAQVq5cibi4OLRv3x4pKSnYuXMnypUrp/aTX7hwAfXr15fuDxs2DADg5eWFoKAgPH/+XAo6AODo6Ii9e/di6NChWLBgAezs7LBixQoeBk5ERESSbIebli1b4tixY2jevDn8/f3RpEkT6OrqIiAgIMdPXq9ePQghPrk8q7MP16tXD5cvX87xcxIREZG8ZTvc7Nu3D4MGDUL//v1RqlSp3KyJiIiIKMeyPefmxIkTiI+PR5UqVVCtWjUsWrQIL1++zM3aiIiIiNSW7XDj5uaG5cuX4/nz5/D29samTZtQpEgRKJVKhIaGIj4+PjfrJCIiIsoWtY+WypcvH3r27IkTJ07g+vXrGD58OGbOnAlra2u0atUqN2okIiIiyrYcHwoOAKVLl8bs2bPx9OlTbNy4UVM1EREREeXYZ4WbDLq6uvDw8MDu3bs18XBEREREOaaRcENERESUVzDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGs5Ilws3jxYjg4OMDIyAjVqlXDuXPnPtk3KCgICoVC5WZkZPQFqyUiIqK8TOvhZvPmzRg2bBgmTpyIS5cuoVKlSnB3d0d0dPQn1zE1NcXz58+lW0RExBesmIiIiPIyrYeb+fPno0+fPujRowfKlSuHgIAAmJiYYNWqVZ9cR6FQwNbWVrrZ2Nh8wYqJiIgoL9NquElNTcXFixfRqFEjqU1HRweNGjXC6dOnP7leQkICihcvDnt7e7Ru3Ro3b978ZN+UlBTExcWp3IiIiEi+tBpuXr58ifT09EwjLzY2NoiMjMxyndKlS2PVqlXYtWsX/vjjDyiVStSoUQNPnz7Nsr+vry/MzMykm729vcZfBxEREeUdWt8tpa7q1avD09MTLi4uqFu3LoKDg2FlZYVly5Zl2X/MmDGIjY2Vbk+ePPnCFRMREdGXpKfNJy9UqBB0dXURFRWl0h4VFQVbW9tsPYa+vj5cXV0RHh6e5XJDQ0MYGhp+dq1ERET0ddDqyI2BgQGqVKmCQ4cOSW1KpRKHDh1C9erVs/UY6enpuH79OgoXLpxbZRIREdFXRKsjNwAwbNgweHl5oWrVqvjhhx/g7++PxMRE9OjRAwDg6emJokWLwtfXFwAwZcoUuLm5wcnJCTExMZgzZw4iIiLQu3dvbb4MIiIiyiO0Hm46dOiAFy9eYMKECYiMjISLiwv2798vTTJ+/PgxdHT+N8D05s0b9OnTB5GRkbCwsECVKlVw6tQplCtXTlsvgYiIiPIQrYcbABgwYAAGDBiQ5bIjR46o3Pfz84Ofn98XqIqIiIi+Rl/d0VJERERE/4bhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkJU+Em8WLF8PBwQFGRkaoVq0azp0796/9t27dijJlysDIyAgVK1bEX3/99YUqJSIiorxO6+Fm8+bNGDZsGCZOnIhLly6hUqVKcHd3R3R0dJb9T506hU6dOqFXr164fPkyPDw84OHhgRs3bnzhyomIiCgv0nq4mT9/Pvr06YMePXqgXLlyCAgIgImJCVatWpVl/wULFqBJkyYYOXIkypYti6lTp6Jy5cpYtGjRF66ciIiI8iKthpvU1FRcvHgRjRo1ktp0dHTQqFEjnD59Ost1Tp8+rdIfANzd3T/Zn4iIiL4tetp88pcvXyI9PR02NjYq7TY2Nrh9+3aW60RGRmbZPzIyMsv+KSkpSElJke7HxsYCAOLi4j6n9E9SprzNlcelr0dubVvZxW2QuA2StuXGNpjxmEKI/+yr1XDzJfj6+mLy5MmZ2u3t7bVQDX0LzPy1XQF967gNkrbl5jYYHx8PMzOzf+2j1XBTqFAh6OrqIioqSqU9KioKtra2Wa5ja2urVv8xY8Zg2LBh0n2lUonXr1+jYMGCUCgUn/kK6ENxcXGwt7fHkydPYGpqqu1y6BvEbZC0jdtg7hFCID4+HkWKFPnPvloNNwYGBqhSpQoOHToEDw8PAO/Dx6FDhzBgwIAs16levToOHTqEIUOGSG2hoaGoXr16lv0NDQ1haGio0mZubq6J8ukTTE1N+UtNWsVtkLSN22Du+K8Rmwxa3y01bNgweHl5oWrVqvjhhx/g7++PxMRE9OjRAwDg6emJokWLwtfXFwAwePBg1K1bF/PmzUPz5s2xadMmXLhwAYGBgdp8GURERJRHaD3cdOjQAS9evMCECRMQGRkJFxcX7N+/X5o0/PjxY+jo/O+grho1amDDhg0YP348xo4di1KlSmHnzp2oUKGCtl4CERER5SEKkZ1px0TZkJKSAl9fX4wZMybTrkCiL4HbIGkbt8G8geGGiIiIZEXrZygmIiIi0iSGGyIiIpIVhhsiIiKSFYYbIiIikhWGGyIiIpIVhhsiIiKSFYYbIiIikhWGGyIiIpIVhhsiIiKSFYYb+mYolUptl0BERF8Aww19MzIuwPry5UsAAK88Ql/axwGb2yBpw8fboRy/+DHc0DdlwYIF8PDwwP3796FQKLRdDn1jdHR0EBsbi5CQEADgNkhaoaOjg5iYGMyZMwdv3ryRvvjJifxeEdEHPv5mrK+vD2NjYxgYGGipIvqWKZVKzJs3D97e3tizZ4+2y6Fv2IEDBzB//nwsWrRI26XkCl4VnL4JcXFxMDU1BQDExsbCzMxMyxXRt0KpVKp8M7516xZWrlyJWbNmQVdXV4uV0bckPT1dZXtLS0vD5s2b0alTJ1luhww3JHtDhw5Feno6xowZg8KFC2u7HPoGxcTEICYmBvb29iofJB9/4BB9jo+D9MdevXqFkydPokaNGihUqJDULsftkLulSHY+zut2dnZYu3at7H556esghMCvv/6KatWq4dGjRyrLuE3S53j+/Dn++ecfvHjxAsD7uTT/Nl6xZcsWeHh44OjRoyrtctwOOXJDX7WMbxxCCCgUik9+c3nz5g0sLCy0UCHJzX99O86qT0REBMaPH4+goCBZfpDQl7d69WosXrwYT548QcmSJVGrVi3Mnj1bpU9WIzL+/v4YMGAA9PT0vmS5XxzDDX01MgIM8P6XVggBPT09PHv2DDt27ECPHj2QL18+AO93RVlYWGDChAmZ1iXKqQ9Dy+HDh/H48WM4OTmhRIkSKFKkiEqf2NhYKJXKTKFajrsA6Mvas2cP2rdvjyVLlsDExAQPHjzA7NmzUaNGDaxZswYFCxaU/ua9fPkS4eHhcHNzU3mMd+/eyTrgcLcU5VkZuTsuLg5JSUlQKBQ4cOAAwsPDoaurCz09PURERMDV1RX//POPFGwSExOhr68PPz8/vH79msGGNEIIIQWbX3/9Fd27d8fcuXPRt29fjBgxAufPnwfwftdASkoKJkyYgMqVK+PVq1cqj8NgQ5/r/PnzaN68Obp374727dtj1KhRCAkJwbVr19ClSxcA708zkJaWhnXr1qFGjRo4ceKEymPIOdgADDeUx0VGRqJixYo4evQoNmzYgCZNmuDvv/8G8H5XU/ny5dGmTRtMnz5dWidfvnwYNWoU7t27B0tLSwYb0oiM7Wju3Ln4448/sHHjRty4cQNt27bFn3/+ifHjx+P06dMAAAMDA7i6uqJhw4YwNzfXYtUkRw8fPsTz589V2r7//nvs3r0bFy9eRJ8+fQC8P/VFixYtMH369EwjN7IniPK4Hj16CFNTU6GjoyOWL18utaemporNmzeL9PR0qU2pVGqjRPpGREVFibZt24pVq1YJIYTYvXu3MDU1Ff369ROurq6iYcOG4syZM0II1W3x3bt3WqmX5CkkJETY2NiITZs2SW0Z29v69euFk5OTOH/+fKb10tLSvliN2saRG8qzMk4J7uPjg/j4eBgYGMDW1hbJyckA3n8rad++vcrETY7SUG6ytrbGqFGj0KRJE1y+fBk+Pj6YNm0ali5dip9++glnzpyBj48PLl68qLItclcUaVLZsmVRr149rFu3DocOHQLwv799Li4uiI6Oli4z8yG574r6EMMN5VkZocXe3h4nTpyAl5cXOnbsiF27diEpKSlTfzleH4W051Pbk6urKwoXLox9+/bB2dkZffv2BQBYWlrCzc0NLVu2hKur65cslb4x9vb26NevH2JiYuDn54fdu3dLywoXLgxHR0ctVpc3fDsxjr4a4v8nAD9//hxpaWkoVqwYrK2tUaNGDSQnJ6NXr14ICgpCixYtYGRkhICAADRq1AhOTk7aLp1kQnwweXjFihWIjo6GgYEBRowYIV26IyUlBc+ePcOjR49QunRpHDhwAK1atcLAgQP/9bQERJ8j42i7evXqYcmSJRg7dixGjx6NkJAQODs7Y8uWLVAoFPjxxx+1XapW8VBwypOCg4MxadIkREVFoXnz5mjTpg1atmwJAOjRowd27NiB4cOHIyoqCkuXLsX169dRrlw5LVdNcjNx4kT4+/vj+++/x7lz51CtWjWsW7cOtra2+PPPPzFt2jS8efMG+vr6EELg2rVr0NPT4xF6lCsytqvg4GAsWbIEBw4cwO3btxEWFoZFixbB3t4e5ubmWL9+PfT19b/p0w4w3FCec/PmTbi7u2Po0KEwMTHBxo0bYWhoCC8vL3Tt2hUAMHjwYFy6dAkpKSkIDAyEi4uLdosmWfhwtOXdu3fw8vLCwIED4erqikePHqF58+awtbXFjh07YGVlhb179yI8PBwJCQkYPXo09PT0vukPFNKMjBAjPjq3l66uLoKDg+Hp6Yn58+dLu0SB99urjo6Oyvb7Lc2x+RjDDeUpt2/fxtatW5GUlIQZM2YAAK5fv44JEyYgLi4OPXr0kAJOZGQk8uXLhwIFCmizZJKJD4PNrVu3EBcXh2XLlmHChAlwcHAA8P4Q3B9//BE2NjbYuXMnrKysVB6DwYY+14fb4cuXL6FQKFCwYEEA7//mVa5cGRMmTEC/fv2kdT4eKeTIIcMN5RFCCLx58wYtWrTA33//jZYtW2LdunXS8mvXrmHChAlISkpCx44d0aNHDy1WS3I2cuRIaVg/KioKwcHBaNq0qfRh8fDhQzRt2hRCCJw8eVLlAoREn+PDUDJ16lTs3LkTcXFxKFSoEKZPn44GDRrg2bNnKFq0qJYrzfs4243yBIVCAUtLS/j6+qJ8+fK4dOkSQkNDpeXOzs6YOnUq0tLSpF94Ik348KioPXv2YP/+/fj999+xZMkSODo6Yty4cbh69ap0xmxHR0fs2bMHLi4uvF4ZaVRGsJkyZQoWLFggnWqgUKFC6NKlC9asWZNptJCyxpEb0ppPDZ0ePXoUY8eOha2tLXx8fNCgQQNp2c2bN2FmZgY7O7svWSp9A4KDg3Hq1CkULFgQY8aMAQAkJCSgcuXKMDU1xYoVK1CpUqVM2yx3RZEmvXr1Co0bN4aPjw969uwptfft2xd//vknwsLCUKZMGe56+g8cuSGtyPjFPHXqFObPn4/ffvsNJ0+eRFpaGurWrYspU6YgMjISixYtwpEjR6T1ypcvz2BDGpeUlITffvsN8+fPx82bN6X2/Pnz49KlS4iPj4e3t7d0/agPMdiQJr179w4vX76URgUzTloaGBiIIkWKwM/PDwBPWPpfGG7oi/vwcMamTZvi5MmT2L17N8aOHYvp06cjNTUVDRs2xJQpU/Dq1StMnToVx48f13bZJGPGxsY4fvw4GjVqhIsXL2L37t1IT08H8L+Ac/v2bSxbtkzLlZKcZLXjxMbGBra2tli1ahUAwMjICKmpqQAAJycnhppsYrihLy5jxGbQoEGYP38+tm/fjq1bt+LixYvYvHkzxo8fLwWcX3/9Ffr6+jzjJmnMh3NshBDSB4ylpSU2bNgACwsLzJkzByEhIdKyfPnyITIyEoGBgVqpmeRHqVRKQeWff/5BdHQ03r59CwCYNGkSbt++LR0RlXHiyKdPn/JCrNnEOTf0xWT8MisUCixZsgRXrlxBYGAgHj58iEaNGqFWrVowNTXF1q1b4e3tjbFjx8LQ0BBv376FiYmJtssnGfjwMNuFCxfi6tWrePDgAYYMGYLKlSvDzs4OL168QOvWraGrq4uxY8fC3d1d5UzDnGNDn2P9+vVwc3NDyZIlAQBjxoxBSEgIIiIi0KhRI7Rq1QpdunTB8uXLMXXqVBQsWBAVKlTA/fv3ERMTI50okv4dww3lmowPkg/DyZUrV+Di4oK4uDg8efIETk5OaNKkCRwdHbFq1SrExsZKZxru3r07pk+fzolz9Nk+3obGjBmDlStXom/fvnj69ClOnz6N1q1bo2/fvnBycsKLFy/Qtm1bvHjxAkFBQXBzc9Ni9SQX+/btQ4sWLTB69GgMGTIE+/btw6hRo+Dv749Xr17h0qVLCAkJwW+//YZ+/frh+vXr8Pf3h46ODiwsLDBjxgyeKDK7cvWa4/TNe/DggejUqZP4+++/xZYtW4RCoRDnzp0TSqVSCCHE9evXRZkyZcTZs2eFEELcv39ftGjRQowdO1Y8fvxYm6WTzKSnpwshhFi3bp1wdHQUFy9eFEIIcfz4caFQKESpUqXE4MGDxYMHD4QQQjx//lz07dtXvHv3Tms1k/wsWrRI2NnZialTp4oBAwaI5cuXS8uePHkipkyZIhwcHMT+/fuzXD8tLe1LlfpV49gW5ark5GQcP34c3bt3x5UrV7B69Wp8//330i4qIQTevXuH06dPo3z58li7di0AYMSIETyHCH22bt26wcrKCvPnz4eOjg7S0tJgYGCAfv36oXLlyti5cyd69OiBFStWIDIyEtOmTYOOjg769OmDsmXLShOI+U2ZPldqaioMDAzg4+MDExMTjBkzBvHx8Zg2bZrUx87ODp6enjhw4AAuXLgAd3f3TBdg5S6pbNJ2uiL5yvimHBAQIHR0dESlSpXE5cuXVfrExsaK7t27i5IlSwoHBwdhZWUlfaMm+hyxsbFi8uTJwtLSUkyaNElqf/bsmYiKihLPnz8XVatWFfPmzZP6FylSRBQuXFgsWLBACCGkEUYiTfH19RXR0dFi/fr1wsTERDRr1kzcvXtXpU+HDh1E27ZttVShPPBoKcoVQgjo6OhACIEiRYpg3rx5ePfuHcaPH48TJ05I/UxNTTF37lwsWbIEEydOxNmzZ1G5cmUtVk5yEB8fD1NTU/Tv3x/jx4+Hv78/Jk6cCAAoUqQIrK2t8fz5c7x580aaT/Ps2TM0btwYEyZMgI+PDwCeS4Q+n/hgWuuaNWswdepU3Lt3D507d4afnx8uXbqEgIAA3LlzBwAQFxeHhw8folixYtoqWRY4vkUaJ/5/8ubhw4dx9OhRDBkyBC1btkSjRo3Qvn17zJw5E2PHjkWNGjUAvL8wZuPGjbVcNcnFqFGjsGzZMty/fx9WVlbo2rUrhBCYOnUqAGDy5MkA3gcgXV1dnDx5EkIIzJw5EyYmJtLht9wVRZqQEZAPHTqEy5cvIzAwUPrb17dvX6SlpWHy5MnYv38/KleujMTERKSmpmL27NnaLPvrp81hI5KfjGH8bdu2CTMzMzFmzBhx/vx5afm1a9dEuXLlRIsWLcQff/whJk2aJBQKhXjy5Al3AZBGXL16VdSpU0eULl1avHjxQgghRHR0tJg3b54wNzcXEyZMkPoOGDBAlCxZUtjZ2Qk3NzeRmpoqhODuKNKsI0eOiIoVK4qCBQuKnTt3CiGESElJkZavXLlS5M+fX1SuXFmsXbtWmsTOycM5x0PBSePOnTuHJk2aYNasWejTp4/UHhcXB1NTU9y6dQt9+vRBUlISYmNjsWXLFu6KIo04ffo0Xrx4gXLlyqFDhw5ISEiQrtz94sULrFu3DlOnTpUuSAi8Pz2BQqFAxYoVoaOjg3fv3nHSJn0W8dGpBxISEjBnzhwEBgaiWrVq2LhxI4yNjZGWlgZ9fX0AwPz583Hq1Cls3boVCoWCI4efieGGNG7RokXYsWMHDh06hNjYWBw+fBh//PEHbt26hREjRqBnz56Ijo5GbGwszMzMYG1tre2SSSY8PT3xzz//4ODBg3j06BF+/vlnxMfHZwo406ZNw4ABAzBlyhSV9fmBQpq0ePFi2NnZoXXr1khKSsLcuXOxY8cO1KtXDzNmzICRkZFKwMkIRR+HI1IfJxSTxtna2uLixYvw9fXFzz//jNWrV8PIyAjNmzdH7969cffuXVhbW6NUqVIMNqRRixcvxtOnT7Fo0SI4ODhg48aNMDMzQ82aNfHy5UtYWVmhW7dumDBhAqZNm4aVK1eqrM9gQ5ry4sULHD58GL/88gv2798PY2NjDBs2DC1atMCpU6cwbtw4JCcnQ19fH+/evQMABhsN4sgNfZaMX8SEhATkz58fABAVFYWFCxdiy5YtaNCgAbp3744ffvgBUVFRaNWqFYKCglC+fHktV05ykzHq8vvvv+Py5cuYP38+LCwscPv2bXh6eiI2NlYawYmMjMTRo0fx008/cRcUacTH56MBgKtXr+L333/HwYMHERAQgKZNmyIxMRGzZ8/GwYMHUbZsWSxZskS6dhRpDkdu6LMoFArs3bsXnTp1Qr169RAUFAQ9PT1MmzYNZ8+eRUBAANzc3KCjo4OFCxciMTGRozWUKzJGXerVq4djx45h7969AIDSpUtj3bp1sLCwQJ06dRAVFQVbW1t06NABenp60rdmos+REWwiIyOltkqVKmHw4MGoX78++vXrh/379yNfvnwYNWoUfvjhB+jo6Ei7pEjDtDSRmWTi5MmTwsjISIwcOVI0adJEODs7C29vbxEeHi71CQsLE3379hWWlpaZTuJHlFMZJ4nMSkBAgPjuu+/EnTt3pLY7d+4IBwcH0bFjxy9RHn0jPtwON23aJEqUKKFyhKgQQly5ckW0bt1aFCtWTBw5ckQIIURSUpJ0VN6/bcuUMxy5oRyLiIhAaGgopk+fjtmzZ2Pfvn3o27cvrl27Bl9fXzx48ACJiYk4ffo0oqOjcfToUbi4uGi7bJKBD3cBnDt3DqdOncLRo0el5a1atUK1atUQFhYmtX333Xc4duwY/vjjjy9eL8lTSkqKtB2mpqaiZMmSKFOmDHx8fHDx4kWpX6VKleDh4YEnT56gcePGOHXqFIyMjKQ5Nh/vzqLPx58oZcuiRYvw119/Sffv3LmDDh06YNWqVTAyMpLafXx80KVLF9y8eROzZ89GTEwMRo4ciTVr1qBChQraKJ1k5sMPg7Fjx6J79+7o2bMnvLy80KFDB8TFxaFw4cLSfIa0tDRpXXt7e+jq6iI9PV1b5ZNM7Nu3D+vWrQMA9OnTBw0aNEDVqlUxfPhw2NrawtvbGxcuXJD6FytWDB07dsS8efNQrVo1qZ2Th3OJtoeOKO97+PCh6Ny5s7h3755K+6+//iqsra1F27ZtpZOlZVi6dKkoXbq0GDRoEE9ERbli7ty5omDBguLs2bMiPT1dzJgxQygUCnHixAmpT82aNYW3t7cWqyS56tSpk3BwcBDu7u6iUKFC4urVq9Kyw4cPCw8PD1GhQgWxb98+8fDhQ+Hh4SGGDx8u9eHV5nMXww1lS2JiohBCiDNnzoht27ZJ7RMmTBAVK1YU48ePF1FRUSrrLF++XDx8+PBLlknfCKVSKby8vERgYKAQQojt27cLc3NzERAQIIQQIj4+XgghxL59+0SrVq3EtWvXtFYryZeLi4tQKBQqF2bNcPz4cdGtWzehUCjEd999J5ydnaUvejwDdu7jMZCULcbGxoiJiYGvry+ePXsGXV1deHh4YPLkyUhLS8PevXshhMDgwYNhZWUFAOjdu7eWqya5Sk5OxtmzZ1GvXj0cOXIEXl5emDNnDry9vfHu3TvMnj0b1atXh5ubG6ZMmYJz586hYsWK2i6bZCI1NRXJyclwcnJCsWLFsHnzZhQtWhQdO3aUTolRq1YtVKtWDX369EFaWhrq1q0LXV1dngH7C+GcG8oWhUIBc3NzDB8+HI6OjvD390dwcDAAYMaMGWjSpAlCQ0MxY8YMvHz5UsvVkpxcu3YNT58+BQAMHToUR48ehbGxMTp37ow//vgDzZo1g5+fn3TByzdv3uDChQu4c+cOLCwssG7dOhQvXlybL4FkxsDAAKampti6dSt27dqF77//HrNnz8amTZsQHx8v9UtOTkbt2rXRoEEDaa4Xg82XwXBD2SLe78JE7dq1MXToUFhYWOD3339XCThubm64fPkyBM8LSRoghMDdu3dRv359rFq1Cv369cOCBQtgYWEBAHBzc0NERASqVauG6tWrAwD++ecfdO/eHTExMRgwYAAAoGTJkmjUqJHWXgfJjxACSqVSur9mzRrUqFEDfn5+WLt2LR4/fowGDRqgXbt2Un+AZ8D+kniGYsqWjLO/xsbGwsTEBNeuXcP06dPx5s0bDB48GB4eHgDen3I8Y7cUkSYsX74co0aNQnJyMnbt2oXGjRtLZ8bevHkzpkyZAiEE9PT0YGxsDKVSiVOnTkFfX5/XiqLP9vr1a1haWqq0ZWx/W7duRWhoKAIDAwEAffv2xZEjR5Ceng5LS0ucPHmSZx/WEo7c0H969+4ddHV18ejRI9SrVw8HDhxAlSpVMGLECFhZWWHy5MnYs2cPADDYkMZkfDO2t7eHoaEhTE1NcebMGTx69Eg6fLZDhw5Yu3YtpkyZgvbt22P06NE4c+aMdL0eBhv6HAsWLMD333+vsqsJgBRsunfvjkqVKkntgYGBWLZsGRYuXIgzZ87AwMCAZ8DWFu3MY6a86lOz+MPDw4WNjY3o3bu3yiGMR44cEd26dROPHj36UiWSzH28DaampoqkpCSxdOlSUbRoUTF27Nj/3N54mC19rmXLlglDQ0OxYcOGTMseP34sKlasKBYtWiS1ZbXNcTvUHu6WIon4/6HW06dP49atWwgPD4enpycKFy6MNWvW4MKFC1izZk2mK9cmJyernMiPKKc+PPPw69evER8frzIZ2N/fH3PnzkWvXr3Qo0cPODg4oGXLlhg3bhzc3Ny0VTbJzPLlyzFw4ECsW7cO7dq1Q0xMDBITE5GcnAxra2sUKFAA9+7dQ6lSpbRdKn0Cww2p2L59O/r27StdYPDFixfo0KEDRo8ejQIFCmi7PJKxD4PNlClTcODAAdy4cQPt27dHmzZt0LRpUwDvA46/vz8qVKiAV69e4fHjx3j06BEvQEga8eDBAzg5OaF9+/bYtGkTbty4gV9++QUvXrxAREQE6tevj/79+6NFixbaLpX+BY9JI8mNGzcwdOhQzJs3D927d0dcXBzMzc1hbGzMYEO5LiPYTJgwAYGBgZgzZw4cHBzQr18/3Lt3DzExMejUqROGDBmCQoUK4erVq0hOTsbx48elq3vzMFv6XFZWVpg1axYmTJiAESNG4MCBA6hduzZat26NuLg4bNu2DePHj0ehQoU4WpiXaXOfGGnP4cOHxf379zO1Va9eXQghxK1bt0Tx4sVF7969peX379/nPmTKVYcPHxbly5cXx44dE0IIcerUKWFgYCDKlSsnqlWrJrZu3Sr1/fCyHrzEB2lScnKymDt3rtDR0RE9e/YUqamp0rILFy6I0qVLi8WLF2uxQvovPFrqGyOEwOXLl9G0aVMsXboUERER0rJnz55BCIGEhAQ0adIEjRs3xrJlywAAoaGhWLp0Kd68eaOt0kmGxEd7xYsWLYr+/fujdu3aOHDgAFq0aIHAwECEhobi/v37+P3337Fy5UoAUBml4YgNaZKhoSH69euH7du3o3fv3tDX15e21SpVqsDIyAhPnjzRcpX0bxhuvjEKhQKurq6YN28etmzZgqVLl+LBgwcAgObNmyMqKgqmpqZo3rw5AgMDpV0FISEhuHbtGg+tJY1RKpXSpPQHDx4gMTERpUqVQqdOnZCcnIwFCxZg0KBB6NatG4oUKYLy5csjPDwct27d0nLl9C3Ily8fmjZtKp0gMmNbjY6OhrGxMcqXL6/N8ug/8OvONyZjXoKPjw8AYM6cOdDV1UXv3r3h6OiI3377DTNmzMC7d+/w9u1bhIeHY+PGjVixYgVOnDghnR2W6HN8OHl4woQJOH36NEaOHIn69evD0tISiYmJeP78OUxMTKCjo4OUlBQ4ODhg1KhRaNKkiZarJzkSHxwBmsHQ0FD6f3p6Ol6+fIk+ffpAoVCgU6dOX7pEUgPDzTcmY+TlwIED0NHRQVpaGvz9/ZGcnIzRo0ejffv2SEpKwowZM7Bt2zbY2NjAwMAAYWFhqFChgparJ7n4MNgsW7YMgYGBcHV1lY54SklJgaWlJU6cOCFNGn716hVWrVoFHR0dlXBElBMRERF4/fo1ChYsCFtb2389k3BaWhrWrVuHjRs34vXr1zhz5ox0rSiOZudNPBT8GxQSEiJdbDBfvny4d+8efv/9d/zyyy8YPXo0rKysEB8fj6NHj8LBwQHW1tawtrbWdtn0lfs4kNy9exceHh6YNWsWWrZsmanf+fPnMX78eCQkJMDS0hLBwcHQ19dnsKHPtnbtWsybNw/R0dEoVKgQBg4cKI3IZPh4OwsNDcXNmzcxYMAAHp33FWC4+cYolUp06dIFCoUCGzZskNoXLlyIUaNGwcfHB7/88gtKlCihxSpJbtq2bYuxY8eiatWqUtuVK1fQpEkTHD16FKVLl87yxJDJyckQQsDIyAgKhYIfKPTZ1q5dCx8fH+nSCjNmzMCDBw9w8uRJadvKCDYxMTE4cOAA2rdvr/IYHLHJ+/j15xuT8U0kY/g/NTUVADBw4EB4e3tj9erV+P3331WOoiL6XGZmZnB2dlZpMzIywps3b3Djxg2pLeN6UqdPn8b27duho6MDY2NjKBQKKJVKBhv6LBcuXMDUqVOxaNEi9OzZExUrVsTQoUPh5OSEU6dO4ebNm4iLi5N22a9Zswa//PIL/vjjD5XHYbDJ+xhuvhH//POP9P/SpUvjzz//RHR0NAwMDJCWlgYAsLOzg4mJCcLCwmBsbKytUklGnj17BgBYvXo1DAwM8Pvvv+PAgQNITU2Fk5MTOnTogDlz5uDgwYNQKBTQ0dFBeno6pk+fjrCwMJV5ENwVRZ8rJSUFQ4YMQfPmzaW2SZMm4dChQ+jUqRM8PT3RsWNHvH79Gvr6+mjWrBlGjBjBycNfIe6W+gZcvXoVAwYMQOfOndG/f3+kpqaiQYMGePnyJY4cOQJbW1sAwOjRo1G+fHm0aNEClpaWWq6avnZ9+vQBAIwZM0bazens7IyXL19i06ZNqFOnDo4fPw4/Pz9cv34dXbp0gYGBAQ4dOoQXL17g0qVLHKkhjVIqlXjx4gVsbGwAAJ6enjh48CB2794Ne3t7HD16FNOmTcPo0aPRuXNnlTk43BX1deFXoW+AiYkJzM3NsW3bNgQFBcHAwADLli2DlZUVypYtCw8PDzRu3BgLFixA1apVGWxII5ydnbF//34sXboU4eHhAIBr166hdOnS6NKlC44dO4batWtjypQp8PT0xLp163D48GEUK1YMFy9elCZtEmmKjo6OFGwAYMSIETh79iyqVq0KGxsbNG3aFK9fv0ZUVFSmw8IZbL4uHLn5RoSHh2Ps2LGIjIxEnz590K1bN6Snp2Pu3LmIiIiAEAIDBw5EuXLltF0qyciqVaswYcIEdOzYEX369EHp0qUBAHXq1MHDhw+xfv161KlTBwDw9u1bmJiYSOty8jB9aU+fPkXXrl0xYsQIXhjzK8dwI1OXLl3C8+fPVfYth4eHY/z48Xj06BEGDhyILl26aLFCkrMPD6NduXIlJkyYgE6dOmUKOBEREVi7di2qV6+uMr8mqxOqEanjw20o4/8Z/7548QJWVlYq/RMTE9GpUyfExsbi8OHDHKn5yjHcyFB8fDyaN28OXV1djBo1Ck2bNpWWPXr0CE2aNIGJiQl69+6NX375RYuVktx86hw0y5cvx+TJk9GhQwf07dtXCjgNGjTAyZMncebMGbi6un7pckmmstoOM9qCg4OxceNGLFiwAEWKFEFSUhJ27dqFdevW4dmzZzh//jz09fU5x+Yrxzk3MpKRUwsUKIDZs2dDT08PixYtwt69e6U+Dg4OqF+/PiIjI3Ho0CHExMRoqVqSmw8/UE6dOoWwsDBcvXoVwPvJxb/99hs2bdqEwMBA3LlzBwBw+PBh9O7dO9Nh4kQ5deLECemilsOGDcPMmTMBvJ9vs3nzZnh6eqJRo0YoUqQIgPcXXX348CFKlCiBCxcuQF9fH+/evWOw+cpx5EYGMoZaM75pZHzInD17Fr/++ivy5cuH/v37S7uohg8fjhIlSqBt27YoXLiwlqsnOfhwF8CwYcOwefNmJCQkwM7ODsWKFcO+ffsAAMuWLcO0adPQsWNHeHl5qVzSg9+U6XMIIRAbGwtra2s0bdoUhQoVQnBwMI4fP44KFSogJiYGbm5u8PHxwcCBA6V1PvzbCXA7lAuGm69cxi9nWFgYdu/ejdevX6NWrVpo164dzM3NcebMGfz2229ISUlBiRIlYGJigs2bN+Pq1auws7PTdvkkAx8GmwMHDmDIkCEIDAyEubk5/v77b0ycOBH58uXDhQsXALyfg+Pt7Q1/f38MGDBAm6WTDEVHR6NEiRJIT0/H9u3b0axZM2lZVnNtspqbQ18/7pb6yikUCuzYsQMtW7bE27dv8fbtW6xbtw79+/fH69ev4ebmhrlz56Ju3boIDw/HgwcPcPjwYQYb0piMD4Pdu3dj06ZNaNSoEWrVqoUKFSrg559/xtq1a5GQkID+/fsDAHr16oVdu3ZJ94k0JSUlBZGRkTAxMYGuri5WrVolnYYAAAoVKiT9P+Ns2B+GGQYb+eDIzVfuwoUL6NixI3799Vf07t0bERERqFy5MoyNjeHi4oK1a9fC0tJSulbPx4fbEmnC69ev0aJFC1y9ehX169fHnj17VJaPHTsWJ0+exF9//YV8+fJJ7dwFQJ/rU5PYHz16BGdnZ9SvXx/z589HyZIltVAdaQtHbr4ivr6+GDdunPSNA3h/ens3Nzf07t0bjx49QsOGDeHh4YHx48fj/Pnz+OWXX/D69WsYGRkBAIMNacSH2yAAWFpaYs2aNfjxxx9x+fJlrF69WmV5qVKl8OrVKyQlJam0M9jQ5/gw2Bw5cgQbNmzA1atX8ezZMzg4OODkyZMICwvDqFGjpEnsbdq0wcKFC7VZNn0BHLn5iixcuBCDBw/GjBkzMGrUKOmX+tatWyhdujRat24tfcgolUq4uLggPDwczZs3x+bNm3ltHtKIDz9Q7t+/D4VCARMTE9ja2uLhw4fw8fFBYmIi2rVrB29vb0RFRcHLywtGRkbYs2cPh/5J40aMGIE1a9ZAT08P+fPnh62tLfz8/FC1alVcv34d9evXh4ODA1JTU/Hu3TtcvXpVungwyZSgr4JSqRRCCLF8+XKho6Mjpk6dKtLS0qTlT548EWXLlhV79uwRQgjx+vVr0alTJ7Fw4ULx9OlTrdRM8pOxHQohxMSJE0XFihVFmTJlROHChUVgYKAQQojw8HDRrFkzYWRkJEqXLi3atGkj3N3dRVJSkhBCiPT0dK3UTvLx4XYYGhoqKlWqJI4fPy5ev34tdu3aJdq0aSOcnJzEpUuXhBBC3Lt3T0yZMkVMnz5d+rv54d9Pkh+Gm6+AUqmUfpmVSqX4448/hI6Ojpg2bZr0QREdHS1cXFyEt7e3ePTokRg7dqz4/vvvRVRUlDZLJ5maMmWKsLKyEiEhISIhIUG0adNGmJubi5s3bwohhHjw4IFo3ry5cHFxEX5+ftJ6ycnJWqqY5GjNmjViwIABom/fvirt58+fF02aNBFeXl4iISFBCKEaiBhs5I/7Kb4SCoUCBw8exPDhw1GlShXpmj0zZ86EEAIWFhbo0qULjh49Cjc3N6xduxYBAQGwtrbWdukkAx/OsVEqlTh37hz8/PzQuHFjhIaG4siRI5gxYwbKlSuHtLQ0ODo6Yt68ebCxscHevXsRHBwMADA0NNTWSyAZEB/Noti5cycWL16MK1euICUlRWqvWrUqateujRMnTiA9PR2A6pFQvGbZN0Db6YqyZ/v27cLY2FhMnTpVnD9/XgghRGBgoLSLSgghUlJSxM2bN0VoaKh48uSJNsslmZowYYKYOXOmKFq0qLhz544ICwsT+fPnF0uXLhVCCPH27Vsxbtw48ejRIyGEEHfv3hUtWrQQVatWFcHBwdosnb5yH468rF+/Xqxdu1YIIcSAAQOEubm5WLx4sYiNjZX6hISEiDJlykjbIn1bGG6+Anfu3BGOjo5iyZIlmZYtW7ZM2kVFpGkfzo/ZtGmTsLe3Fzdu3BBdu3YV7u7uwsTERKxcuVLq8+zZM1G7dm2xdu1aad1bt26Jn3/+WURERHzx+kkePtwOb9y4IVxdXUWlSpXErl27hBBCeHl5iVKlSonp06eL8PBwER4eLho2bCjq1q2rEoro28Gxua/A48ePoa+vr3KmzYwjVvr27Yt8+fKhW7duMDQ0xIgRI7RYKclNxlFRR48exZEjRzB8+HCUL19eOjlkw4YN0bNnTwDvL9jau3dv6OrqonPnztDR0YFSqUSZMmWwYcMGHp1COZaxHY4cORIPHz6EsbExbt++jaFDh+Ldu3cICgpCz549MX78eCxcuBA1a9ZE/vz5sXnzZigUik+eC4fki+HmK5CQkKByfhClUintPz5y5AiqVKmCzZs3q1ynh0hTIiMj0atXL0RHR2Ps2LEAgH79+uH+/fs4fPgwXF1dUapUKTx+/BjJyck4f/48dHV1VU7QxzkO9LmCgoKwYsUKHDp0CI6OjkhJSYGXlxd8fX2ho6ODVatWwcTEBFu2bEGTJk3QsWNHGBoaIjU1FQYGBtoun74wRtmvQKVKlfDy5UsEBgYCeP8tJiPc7Nq1Cxs2bEDbtm1RtmxZbZZJMmVra4vg4GDY2Njgzz//xMWLF6Grq4s5c+ZgypQpaNCgAWxtbdGhQ4dPXlWZ57ahzxUeHo4KFSrAxcUFZmZmsLW1xapVq6Crq4uhQ4dix44dWLRoERo1aoT58+dj9+7diI+PZ7D5RvHr1FfA0dERixYtQr9+/ZCWlgZPT0/o6uoiKCgIQUFBOH36NM/0SrnK2dkZ27dvh5eXFwICAjBw4EA4OzujVatWaNWqlUrf9PR0jtSQxoj/v5iloaEhkpOTkZqaCiMjI6SlpaFo0aLw9fVFixYt4O/vD2NjY2zYsAGdO3fGiBEjoKenh/bt22v7JZAW8AzFXwmlUont27fD29sb+fLlg5GREXR1dbFx40a4urpquzz6Rly+fBm9e/dGlSpVMHjwYJQvX17bJdE34vr163B1dcVvv/2GiRMnSu0hISFYvnw53rx5g/T0dBw5cgQA0KNHD/z2228oUaKEliombWK4+cr8888/iIiIgEKhgKOjI2xsbLRdEn1jLl++DG9vbxQvXhyzZ8+Go6Ojtkuib0RQUBD69u2LIUOGoEOHDrCwsMCgQYNQo0YNtGnTBuXLl8fevXvRtGlTbZdKWsZwQ0RqO3fuHAICArBixQoehUJf1Pbt2/HLL7/AwMAAQghYW1vj1KlTiIqKwo8//oht27bB2dlZ22WSljHcEFGOZMyF4GG29KU9e/YMT548QVpaGmrWrAkdHR2MGTMGO3fuRFhYGGxtbbVdImkZww0R5VhGwCHSlps3b2LWrFn466+/cPDgQbi4uGi7JMoDeEgDEeUYgw1p07t375Camgpra2scPXqUE9xJwpEbIiL6qqWlpfEM2KSC4YaIiIhkhbMAiYiISFYYboiIiEhWGG6IiIhIVhhuiIiISFYYbohI9o4cOQKFQoGYmJhsr+Pg4AB/f/9cq4mIcg/DDRFpXffu3aFQKNCvX79My3x8fKBQKNC9e/cvXxgRfZUYbogoT7C3t8emTZuQlJQktSUnJ2PDhg0oVqyYFisjoq8Nww0R5QmVK1eGvb09goODpbbg4GAUK1YMrq6uUltKSgoGDRoEa2trGBkZoVatWjh//rzKY/3111/47rvvYGxsjPr16+PRo0eZnu/EiROoXbs2jI2NYW9vj0GDBiExMTHXXh8RfTkMN0SUZ/Ts2ROrV6+W7q9atQo9evRQ6TNq1Chs374da9aswaVLl+Dk5AR3d3e8fv0aAPDkyRO0bdsWLVu2xJUrV9C7d2/8+uuvKo9x//59NGnSBD/99BOuXbuGzZs348SJExgwYEDuv0giynUMN0SUZ3Tt2hUnTpxAREQEIiIicPLkSXTt2lVanpiYiKVLl2LOnDlo2rQpypUrh+XLl8PY2BgrV64EACxduhQlS5bEvHnzULp0aXTp0iXTfB1fX1906dIFQ4YMQalSpVCjRg38/vvvWLt2LZKTk7/kSyaiXMALZxJRnmFlZYXmzZsjKCgIQgg0b94chQoVkpbfv38faWlpqFmzptSmr6+PH374Abdu3QIA3Lp1C9WqVVN53OrVq6vcv3r1Kq5du4b169dLbUIIKJVKPHz4EGXLls2Nl0dEXwjDDRHlKT179pR2Dy1evDhXniMhIQHe3t4YNGhQpmWcvEz09WO4IaI8pUmTJkhNTYVCoYC7u7vKspIlS8LAwAAnT55E8eLFAby/IvT58+cxZMgQAEDZsmWxe/dulfXOnDmjcr9y5cr4+++/4eTklHsvhIi0hnNuiChP0dXVxa1bt/D3339DV1dXZVm+fPnQv39/jBw5Evv378fff/+NPn364O3bt+jVqxcAoF+/frh37x5GjhyJO3fuYMOGDQgKClJ5nNGjR+PUqVMYMGAArly5gnv37mHXrl2cUEwkEww3RJTnmJqawtTUNMtlM2fOxE8//YRu3bqhcuXKCA8PR0hICCwsLAC83620fft27Ny5E5UqVUJAQABmzJih8hjOzs44evQo7t69i9q1a8PV1RUTJkxAkSJFcv21EVHuUwghhLaLICIiItIUjtwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGs/B+XLE52CERTBAAAAABJRU5ErkJggg==",
+ "text/plain": [
+ "
+
+
+ Bar Graph Excel Sheet here
+
+
+| Model | Provider | Cost per input + output token ($)|
+| --- | --- | --- |
+| openrouter/mistralai/mistral-7b-instruct | openrouter | 0.0 |
+| ollama/llama2 | ollama | 0.0 |
+| ollama/llama2:13b | ollama | 0.0 |
+| ollama/llama2:70b | ollama | 0.0 |
+| ollama/llama2-uncensored | ollama | 0.0 |
+| ollama/mistral | ollama | 0.0 |
+| ollama/codellama | ollama | 0.0 |
+| ollama/orca-mini | ollama | 0.0 |
+| ollama/vicuna | ollama | 0.0 |
+| perplexity/codellama-34b-instruct | perplexity | 0.0 |
+| perplexity/llama-2-13b-chat | perplexity | 0.0 |
+| perplexity/llama-2-70b-chat | perplexity | 0.0 |
+| perplexity/mistral-7b-instruct | perplexity | 0.0 |
+| perplexity/replit-code-v1.5-3b | perplexity | 0.0 |
+| text-bison | vertex_ai-text-models | 0.00000025 |
+| text-bison@001 | vertex_ai-text-models | 0.00000025 |
+| chat-bison | vertex_ai-chat-models | 0.00000025 |
+| chat-bison@001 | vertex_ai-chat-models | 0.00000025 |
+| chat-bison-32k | vertex_ai-chat-models | 0.00000025 |
+| code-bison | vertex_ai-code-text-models | 0.00000025 |
+| code-bison@001 | vertex_ai-code-text-models | 0.00000025 |
+| code-gecko@001 | vertex_ai-chat-models | 0.00000025 |
+| code-gecko@latest | vertex_ai-chat-models | 0.00000025 |
+| codechat-bison | vertex_ai-code-chat-models | 0.00000025 |
+| codechat-bison@001 | vertex_ai-code-chat-models | 0.00000025 |
+| codechat-bison-32k | vertex_ai-code-chat-models | 0.00000025 |
+| palm/chat-bison | palm | 0.00000025 |
+| palm/chat-bison-001 | palm | 0.00000025 |
+| palm/text-bison | palm | 0.00000025 |
+| palm/text-bison-001 | palm | 0.00000025 |
+| palm/text-bison-safety-off | palm | 0.00000025 |
+| palm/text-bison-safety-recitation-off | palm | 0.00000025 |
+| anyscale/meta-llama/Llama-2-7b-chat-hf | anyscale | 0.0000003 |
+| anyscale/mistralai/Mistral-7B-Instruct-v0.1 | anyscale | 0.0000003 |
+| openrouter/meta-llama/llama-2-13b-chat | openrouter | 0.0000004 |
+| openrouter/nousresearch/nous-hermes-llama2-13b | openrouter | 0.0000004 |
+| deepinfra/meta-llama/Llama-2-7b-chat-hf | deepinfra | 0.0000004 |
+| deepinfra/mistralai/Mistral-7B-Instruct-v0.1 | deepinfra | 0.0000004 |
+| anyscale/meta-llama/Llama-2-13b-chat-hf | anyscale | 0.0000005 |
+| amazon.titan-text-lite-v1 | bedrock | 0.0000007 |
+| deepinfra/meta-llama/Llama-2-13b-chat-hf | deepinfra | 0.0000007 |
+| text-babbage-001 | text-completion-openai | 0.0000008 |
+| text-ada-001 | text-completion-openai | 0.0000008 |
+| babbage-002 | text-completion-openai | 0.0000008 |
+| openrouter/google/palm-2-chat-bison | openrouter | 0.000001 |
+| openrouter/google/palm-2-codechat-bison | openrouter | 0.000001 |
+| openrouter/meta-llama/codellama-34b-instruct | openrouter | 0.000001 |
+| deepinfra/codellama/CodeLlama-34b-Instruct-hf | deepinfra | 0.0000012 |
+| deepinfra/meta-llama/Llama-2-70b-chat-hf | deepinfra | 0.0000016499999999999999 |
+| deepinfra/jondurbin/airoboros-l2-70b-gpt4-1.4.1 | deepinfra | 0.0000016499999999999999 |
+| anyscale/meta-llama/Llama-2-70b-chat-hf | anyscale | 0.000002 |
+| anyscale/codellama/CodeLlama-34b-Instruct-hf | anyscale | 0.000002 |
+| gpt-3.5-turbo-1106 | openai | 0.000003 |
+| openrouter/meta-llama/llama-2-70b-chat | openrouter | 0.000003 |
+| amazon.titan-text-express-v1 | bedrock | 0.000003 |
+| gpt-3.5-turbo | openai | 0.0000035 |
+| gpt-3.5-turbo-0301 | openai | 0.0000035 |
+| gpt-3.5-turbo-0613 | openai | 0.0000035 |
+| gpt-3.5-turbo-instruct | text-completion-openai | 0.0000035 |
+| openrouter/openai/gpt-3.5-turbo | openrouter | 0.0000035 |
+| cohere.command-text-v14 | bedrock | 0.0000035 |
+| gpt-3.5-turbo-0613 | openai | 0.0000035 |
+| claude-instant-1 | anthropic | 0.00000714 |
+| claude-instant-1.2 | anthropic | 0.00000714 |
+| openrouter/anthropic/claude-instant-v1 | openrouter | 0.00000714 |
+| anthropic.claude-instant-v1 | bedrock | 0.00000714 |
+| openrouter/mancer/weaver | openrouter | 0.00001125 |
+| j2-mid | ai21 | 0.00002 |
+| ai21.j2-mid-v1 | bedrock | 0.000025 |
+| openrouter/jondurbin/airoboros-l2-70b-2.1 | openrouter | 0.00002775 |
+| command-nightly | cohere | 0.00003 |
+| command | cohere | 0.00003 |
+| command-light | cohere | 0.00003 |
+| command-medium-beta | cohere | 0.00003 |
+| command-xlarge-beta | cohere | 0.00003 |
+| command-r-plus| cohere | 0.000018 |
+| j2-ultra | ai21 | 0.00003 |
+| ai21.j2-ultra-v1 | bedrock | 0.0000376 |
+| gpt-4-1106-preview | openai | 0.00004 |
+| gpt-4-vision-preview | openai | 0.00004 |
+| claude-2 | anthropic | 0.0000437 |
+| openrouter/anthropic/claude-2 | openrouter | 0.0000437 |
+| anthropic.claude-v1 | bedrock | 0.0000437 |
+| anthropic.claude-v2 | bedrock | 0.0000437 |
+| gpt-4 | openai | 0.00009 |
+| gpt-4-0314 | openai | 0.00009 |
+| gpt-4-0613 | openai | 0.00009 |
+| openrouter/openai/gpt-4 | openrouter | 0.00009 |
+| gpt-4-32k | openai | 0.00018 |
+| gpt-4-32k-0314 | openai | 0.00018 |
+| gpt-4-32k-0613 | openai | 0.00018 |
+
+
+
+## Setup:
+```
+git clone https://github.com/BerriAI/litellm
+```
+cd to `benchmark` dir
+```
+cd litellm/cookbook/benchmark
+```
+
+### Install Dependencies
+```
+pip install litellm click tqdm tabulate termcolor
+```
+
+### Configuration
+In `benchmark/benchmark.py` select your LLMs, LLM API Key and questions
+
+Supported LLMs: https://docs.litellm.ai/docs/providers
+
+```python
+# Define the list of models to benchmark
+models = ['gpt-3.5-turbo', 'togethercomputer/llama-2-70b-chat', 'claude-2']
+
+# Enter LLM API keys
+os.environ['OPENAI_API_KEY'] = ""
+os.environ['ANTHROPIC_API_KEY'] = ""
+os.environ['TOGETHERAI_API_KEY'] = ""
+
+# List of questions to benchmark (replace with your questions)
+questions = [
+ "When will BerriAI IPO?",
+ "When will LiteLLM hit $100M ARR?"
+]
+
+```
+
+## Run LLM-Bench
+```
+python3 benchmark.py
+```
+
+## Expected Output
+```
+Running question: When will BerriAI IPO? for model: claude-2: 100%|āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā| 3/3 [00:13<00:00, 4.41s/it]
+
+Benchmark Results for 'When will BerriAI IPO?':
++-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
+| Model | Response | Response Time (seconds) | Cost ($) |
++=================+==================================================================================+===========================+============+
+| gpt-3.5-turbo | As an AI language model, I cannot provide up-to-date information or predict | 1.55 seconds | $0.000122 |
+| | future events. It is best to consult a reliable financial source or contact | | |
+| | BerriAI directly for information regarding their IPO plans. | | |
++-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
+| togethercompute | I'm not able to provide information about future IPO plans or dates for BerriAI | 8.52 seconds | $0.000531 |
+| r/llama-2-70b-c | or any other company. IPO (Initial Public Offering) plans and timelines are | | |
+| hat | typically kept private by companies until they are ready to make a public | | |
+| | announcement. It's important to note that IPO plans can change and are subject | | |
+| | to various factors, such as market conditions, financial performance, and | | |
+| | regulatory approvals. Therefore, it's difficult to predict with certainty when | | |
+| | BerriAI or any other company will go public. If you're interested in staying | | |
+| | up-to-date with BerriAI's latest news and developments, you may want to follow | | |
+| | their official social media accounts, subscribe to their newsletter, or visit | | |
+| | their website periodically for updates. | | |
++-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
+| claude-2 | I do not have any information about when or if BerriAI will have an initial | 3.17 seconds | $0.002084 |
+| | public offering (IPO). As an AI assistant created by Anthropic to be helpful, | | |
+| | harmless, and honest, I do not have insider knowledge about Anthropic's business | | |
+| | plans or strategies. | | |
++-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
+```
+
+## Support
+**š¤ Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://enterprise.litellm.ai/demo) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.
diff --git a/cookbook/codellama-server/README.MD b/cookbook/codellama-server/README.MD
new file mode 100644
index 00000000..82a7e62f
--- /dev/null
+++ b/cookbook/codellama-server/README.MD
@@ -0,0 +1,153 @@
+# CodeLlama Server: Streaming, Caching, Model Fallbacks (OpenAI + Anthropic), Prompt-tracking
+
+Works with: Anthropic, Huggingface, Cohere, TogetherAI, Azure, OpenAI, etc.
+
+[](https://pypi.org/project/litellm/)
+[](https://pypi.org/project/litellm/0.1.1/)
+
+
+[](https://railway.app/template/HuDPw-?referralCode=jch2ME)
+
+**LIVE DEMO** - https://litellm.ai/playground
+
+## What does CodeLlama Server do
+
+- Uses Together AI's CodeLlama to answer coding questions, with GPT-4 + Claude-2 as backups (you can easily switch this to any model from Huggingface, Replicate, Cohere, AI21, Azure, OpenAI, etc.)
+- Sets default system prompt for guardrails `system_prompt = "Only respond to questions about code. Say 'I don't know' to anything outside of that."`
+- Integrates with Promptlayer for model + prompt tracking
+- Example output
+
+
+
+- **Consistent Input/Output** Format
+ - Call all models using the OpenAI format - `completion(model, messages)`
+ - Text responses will always be available at `['choices'][0]['message']['content']`
+ - Stream responses will always be available at `['choices'][0]['delta']['content']`
+- **Error Handling** Using Model Fallbacks (if `CodeLlama` fails, try `GPT-4`) with cooldowns, and retries
+- **Prompt Logging** - Log successful completions to promptlayer for testing + iterating on your prompts in production! (Learn more: https://litellm.readthedocs.io/en/latest/advanced/
+
+ **Example: Logs sent to PromptLayer**
+
+ 
+
+
+- **Token Usage & Spend** - Track Input + Completion tokens used + Spend/model - https://docs.litellm.ai/docs/token_usage
+- **Caching** - Provides in-memory cache + GPT-Cache integration for more advanced usage - https://docs.litellm.ai/docs/caching/gpt_cache
+
+- **Streaming & Async Support** - Return generators to stream text responses - TEST IT š https://litellm.ai/
+
+## API Endpoints
+
+### `/chat/completions` (POST)
+
+This endpoint is used to generate chat completions for 50+ support LLM API Models. Use llama2, GPT-4, Claude2 etc
+
+#### Input
+
+This API endpoint accepts all inputs in raw JSON and expects the following inputs
+
+- `prompt` (string, required): The user's coding related question
+- Additional Optional parameters: `temperature`, `functions`, `function_call`, `top_p`, `n`, `stream`. See the full list of supported inputs here: https://litellm.readthedocs.io/en/latest/input/
+
+#### Example JSON body
+
+For claude-2
+
+```json
+{
+ "prompt": "write me a function to print hello world"
+}
+```
+
+### Making an API request to the Code-Gen Server
+
+```python
+import requests
+import json
+
+url = "localhost:4000/chat/completions"
+
+payload = json.dumps({
+ "prompt": "write me a function to print hello world"
+})
+headers = {
+ 'Content-Type': 'application/json'
+}
+
+response = requests.request("POST", url, headers=headers, data=payload)
+
+print(response.text)
+
+```
+
+### Output [Response Format]
+
+Responses from the server are given in the following format.
+All responses from the server are returned in the following format (for all LLM models). More info on output here: https://litellm.readthedocs.io/en/latest/output/
+
+```json
+{
+ "choices": [
+ {
+ "finish_reason": "stop",
+ "index": 0,
+ "message": {
+ "content": ".\n\n```\ndef print_hello_world():\n print(\"hello world\")\n",
+ "role": "assistant"
+ }
+ }
+ ],
+ "created": 1693279694.6474009,
+ "model": "togethercomputer/CodeLlama-34b-Instruct",
+ "usage": {
+ "completion_tokens": 14,
+ "prompt_tokens": 28,
+ "total_tokens": 42
+ }
+}
+```
+
+## Installation & Usage
+
+### Running Locally
+
+1. Clone liteLLM repository to your local machine:
+ ```
+ git clone https://github.com/BerriAI/litellm-CodeLlama-server
+ ```
+2. Install the required dependencies using pip
+ ```
+ pip install requirements.txt
+ ```
+3. Set your LLM API keys
+ ```
+ os.environ['OPENAI_API_KEY]` = "YOUR_API_KEY"
+ or
+ set OPENAI_API_KEY in your .env file
+ ```
+4. Run the server:
+ ```
+ python main.py
+ ```
+
+## Deploying
+
+1. Quick Start: Deploy on Railway
+
+ [](https://railway.app/template/HuDPw-?referralCode=jch2ME)
+
+2. `GCP`, `AWS`, `Azure`
+ This project includes a `Dockerfile` allowing you to build and deploy a Docker Project on your providers
+
+# Support / Talk with founders
+
+- [Our calendar š](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
+- [Community Discord š](https://discord.gg/wuPM9dRgDw)
+- Our emails āļø ishaan@berri.ai / krrish@berri.ai
+
+## Roadmap
+
+- [ ] Implement user-based rate-limiting
+- [ ] Spending controls per project - expose key creation endpoint
+- [ ] Need to store a keys db -> mapping created keys to their alias (i.e. project name)
+- [ ] Easily add new models as backups / as the entry-point (add this to the available model list)
diff --git a/cookbook/codellama-server/imgs/code-output.png b/cookbook/codellama-server/imgs/code-output.png
new file mode 100644
index 00000000..67e298bd
Binary files /dev/null and b/cookbook/codellama-server/imgs/code-output.png differ
diff --git a/cookbook/codellama-server/imgs/promptlayer_logging.png b/cookbook/codellama-server/imgs/promptlayer_logging.png
new file mode 100644
index 00000000..26b046ac
Binary files /dev/null and b/cookbook/codellama-server/imgs/promptlayer_logging.png differ
diff --git a/cookbook/codellama-server/main.py b/cookbook/codellama-server/main.py
new file mode 100644
index 00000000..d05d6752
--- /dev/null
+++ b/cookbook/codellama-server/main.py
@@ -0,0 +1,102 @@
+import traceback
+from flask import Flask, request, Response
+from flask_cors import CORS
+import litellm
+from util import handle_error
+from litellm import completion
+import os
+import dotenv
+import time
+import json
+
+dotenv.load_dotenv()
+
+# TODO: set your keys in .env or here:
+# os.environ["OPENAI_API_KEY"] = "" # set your openai key here
+# os.environ["ANTHROPIC_API_KEY"] = "" # set your anthropic key here
+# os.environ["TOGETHER_AI_API_KEY"] = "" # set your together ai key here
+# see supported models / keys here: https://litellm.readthedocs.io/en/latest/supported/
+######### ENVIRONMENT VARIABLES ##########
+verbose = True
+
+# litellm.caching_with_models = True # CACHING: caching_with_models Keys in the cache are messages + model. - to learn more: https://docs.litellm.ai/docs/caching/
+######### PROMPT LOGGING ##########
+os.environ["PROMPTLAYER_API_KEY"] = (
+ "" # set your promptlayer key here - https://promptlayer.com/
+)
+
+# set callbacks
+litellm.success_callback = ["promptlayer"]
+############ HELPER FUNCTIONS ###################################
+
+
+def print_verbose(print_statement):
+ if verbose:
+ print(print_statement)
+
+
+app = Flask(__name__)
+CORS(app)
+
+
+@app.route("/")
+def index():
+ return "received!", 200
+
+
+def data_generator(response):
+ for chunk in response:
+ yield f"data: {json.dumps(chunk)}\n\n"
+
+
+@app.route("/chat/completions", methods=["POST"])
+def api_completion():
+ data = request.json
+ start_time = time.time()
+ if data.get("stream") == "True":
+ data["stream"] = True # convert to boolean
+ try:
+ if "prompt" not in data:
+ raise ValueError("data needs to have prompt")
+ data["model"] = (
+ "togethercomputer/CodeLlama-34b-Instruct" # by default use Together AI's CodeLlama model - https://api.together.xyz/playground/chat?model=togethercomputer%2FCodeLlama-34b-Instruct
+ )
+ # COMPLETION CALL
+ system_prompt = "Only respond to questions about code. Say 'I don't know' to anything outside of that."
+ messages = [
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": data.pop("prompt")},
+ ]
+ data["messages"] = messages
+ print(f"data: {data}")
+ response = completion(**data)
+ ## LOG SUCCESS
+ end_time = time.time()
+ if (
+ "stream" in data and data["stream"] == True
+ ): # use generate_responses to stream responses
+ return Response(data_generator(response), mimetype="text/event-stream")
+ except Exception:
+ # call handle_error function
+ print_verbose(f"Got Error api_completion(): {traceback.format_exc()}")
+ ## LOG FAILURE
+ end_time = time.time()
+ traceback_exception = traceback.format_exc()
+ return handle_error(data=data)
+ return response
+
+
+@app.route("/get_models", methods=["POST"])
+def get_models():
+ try:
+ return litellm.model_list
+ except Exception as e:
+ traceback.print_exc()
+ response = {"error": str(e)}
+ return response, 200
+
+
+if __name__ == "__main__":
+ from waitress import serve
+
+ serve(app, host="0.0.0.0", port=4000, threads=500)
diff --git a/cookbook/community-resources/get_hf_models.py b/cookbook/community-resources/get_hf_models.py
new file mode 100644
index 00000000..8c75a241
--- /dev/null
+++ b/cookbook/community-resources/get_hf_models.py
@@ -0,0 +1,89 @@
+import requests
+
+
+def get_next_url(response):
+ """
+ Function to get 'next' url from Link header
+ :param response: response from requests
+ :return: next url or None
+ """
+ if "link" not in response.headers:
+ return None
+ headers = response.headers
+
+ next_url = headers["Link"]
+ print(next_url)
+ start_index = next_url.find("<")
+ end_index = next_url.find(">")
+
+ return next_url[1:end_index]
+
+
+def get_models(url):
+ """
+ Function to retrieve all models from paginated endpoint
+ :param url: base url to make GET request
+ :return: list of all models
+ """
+ models = []
+ while url:
+ response = requests.get(url)
+ if response.status_code != 200:
+ print(f"Failed to retrieve data. Status code: {response.status_code}")
+ return models
+ payload = response.json()
+ url = get_next_url(response)
+ models.extend(payload)
+ return models
+
+
+def get_cleaned_models(models):
+ """
+ Function to clean retrieved models
+ :param models: list of retrieved models
+ :return: list of cleaned models
+ """
+ cleaned_models = []
+ for model in models:
+ cleaned_models.append(model["id"])
+ return cleaned_models
+
+
+# Get text-generation models
+url = "https://huggingface.co/api/models?filter=text-generation-inference"
+text_generation_models = get_models(url)
+cleaned_text_generation_models = get_cleaned_models(text_generation_models)
+
+print(cleaned_text_generation_models)
+
+
+# Get conversational models
+url = "https://huggingface.co/api/models?filter=conversational"
+conversational_models = get_models(url)
+cleaned_conversational_models = get_cleaned_models(conversational_models)
+
+print(cleaned_conversational_models)
+
+
+def write_to_txt(cleaned_models, filename):
+ """
+ Function to write the contents of a list to a text file
+ :param cleaned_models: list of cleaned models
+ :param filename: name of the text file
+ """
+ with open(filename, "w") as f:
+ for item in cleaned_models:
+ f.write("%s\n" % item)
+
+
+# Write contents of cleaned_text_generation_models to text_generation_models.txt
+write_to_txt(
+ cleaned_text_generation_models,
+ "huggingface_llms_metadata/hf_text_generation_models.txt",
+)
+
+# Write contents of cleaned_conversational_models to conversational_models.txt
+write_to_txt(
+ cleaned_conversational_models,
+ "huggingface_llms_metadata/hf_conversational_models.txt",
+)
diff --git a/cookbook/community-resources/max_tokens.json b/cookbook/community-resources/max_tokens.json
new file mode 100644
index 00000000..289f8faf
--- /dev/null
+++ b/cookbook/community-resources/max_tokens.json
@@ -0,0 +1,93 @@
+{
+ "gpt-3.5-turbo": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002
+ },
+ "gpt-3.5-turbo-0613": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002
+ },
+ "gpt-3.5-turbo-0301": {
+ "max_tokens": 4000,
+ "input_cost_per_token": 0.0000015,
+ "output_cost_per_token": 0.000002
+ },
+ "gpt-3.5-turbo-16k": {
+ "max_tokens": 16000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000004
+ },
+ "gpt-3.5-turbo-16k-0613": {
+ "max_tokens": 16000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000004
+ },
+ "gpt-4": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.00006
+ },
+ "gpt-4-0613": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.00006
+ },
+ "gpt-4-32k": {
+ "max_tokens": 8000,
+ "input_cost_per_token": 0.00006,
+ "output_cost_per_token": 0.00012
+ },
+ "claude-instant-1": {
+ "max_tokens": 100000,
+ "input_cost_per_token": 0.00000163,
+ "output_cost_per_token": 0.00000551
+ },
+ "claude-2": {
+ "max_tokens": 100000,
+ "input_cost_per_token": 0.00001102,
+ "output_cost_per_token": 0.00003268
+ },
+ "text-bison-001": {
+ "max_tokens": 8192,
+ "input_cost_per_token": 0.000004,
+ "output_cost_per_token": 0.000004
+ },
+ "chat-bison-001": {
+ "max_tokens": 4096,
+ "input_cost_per_token": 0.000002,
+ "output_cost_per_token": 0.000002
+ },
+ "command-nightly": {
+ "max_tokens": 4096,
+ "input_cost_per_token": 0.000015,
+ "output_cost_per_token": 0.000015
+ },
+ "replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1": {
+ "max_tokens": 4096,
+ "input_cost_per_token": 0.00000608,
+ "output_cost_per_token": 0.00000608
+ },
+ "together-ai-up-to-3b": {
+ "input_cost_per_token": 0.0000001,
+ "output_cost_per_token": 0.0000001
+ },
+ "together-ai-3.1b-7b": {
+ "input_cost_per_token": 0.0000002,
+ "output_cost_per_token": 0.0000002
+ },
+ "together-ai-7.1b-20b": {
+ "max_tokens": 1000,
+ "input_cost_per_token": 0.0000004,
+ "output_cost_per_token": 0.0000004
+ },
+ "together-ai-20.1b-40b": {
+ "input_cost_per_token": 0.000001,
+ "output_cost_per_token": 0.000001
+ },
+ "together-ai-40.1b-70b": {
+ "input_cost_per_token": 0.000003,
+ "output_cost_per_token": 0.000003
+ }
+}
diff --git a/cookbook/gollem_go_agent_framework/README.md b/cookbook/gollem_go_agent_framework/README.md
new file mode 100644
index 00000000..729f985d
--- /dev/null
+++ b/cookbook/gollem_go_agent_framework/README.md
@@ -0,0 +1,119 @@
+# Gollem Go Agent Framework with LiteLLM
+
+A working example showing how to use [gollem](https://github.com/fugue-labs/gollem), a production-grade Go agent framework, with LiteLLM as a proxy gateway. This lets Go developers access 100+ LLM providers through a single proxy while keeping compile-time type safety for tools and structured output.
+
+## Quick Start
+
+### 1. Start LiteLLM Proxy
+
+```bash
+# Simple start with a single model
+litellm --model gpt-4o
+
+# Or with the example config for multi-provider access
+litellm --config proxy_config.yaml
+```
+
+### 2. Run the examples
+
+```bash
+# Install Go dependencies
+go mod tidy
+
+# Basic agent
+go run ./basic
+
+# Agent with type-safe tools
+go run ./tools
+
+# Streaming responses
+go run ./streaming
+```
+
+## Configuration
+
+The included `proxy_config.yaml` sets up three providers through LiteLLM:
+
+```yaml
+model_list:
+ - model_name: gpt-4o # OpenAI
+ - model_name: claude-sonnet # Anthropic
+ - model_name: gemini-pro # Google Vertex AI
+```
+
+Switch providers in Go by changing a single string ā no code changes needed:
+
+```go
+model := openai.NewLiteLLM("http://localhost:4000",
+ openai.WithModel("gpt-4o"), // OpenAI
+ // openai.WithModel("claude-sonnet"), // Anthropic
+ // openai.WithModel("gemini-pro"), // Google
+)
+```
+
+## Examples
+
+### `basic/` ā Basic Agent
+
+Connects gollem to LiteLLM and runs a simple prompt. Demonstrates the `NewLiteLLM` constructor and basic agent creation.
+
+### `tools/` ā Type-Safe Tools
+
+Shows gollem's compile-time type-safe tool framework working through LiteLLM's tool-use passthrough. The tool parameters are Go structs with JSON tags ā the schema is generated automatically at compile time.
+
+### `streaming/` ā Streaming Responses
+
+Real-time token streaming using Go 1.23+ range-over-function iterators, proxied through LiteLLM's SSE passthrough.
+
+## How It Works
+
+Gollem's `openai.NewLiteLLM()` constructor creates an OpenAI-compatible provider pointed at your LiteLLM proxy. Since LiteLLM speaks the OpenAI API protocol, everything works out of the box:
+
+- **Chat completions** ā standard request/response
+- **Tool use** ā LiteLLM passes tool definitions and calls through transparently
+- **Streaming** ā Server-Sent Events proxied through LiteLLM
+- **Structured output** ā JSON schema response format works with supporting models
+
+```
+Go App (gollem) ā LiteLLM Proxy ā OpenAI / Anthropic / Google / ...
+```
+
+## Why Use This?
+
+- **Type-safe Go**: Compile-time type checking for tools, structured output, and agent configuration ā no runtime surprises
+- **Single proxy, many models**: Switch between OpenAI, Anthropic, Google, and 100+ other providers by changing a model name string
+- **Zero-dependency core**: gollem's core has no external dependencies ā just stdlib
+- **Single binary deployment**: `go build` produces one binary, no pip/venv/Docker needed
+- **Cost tracking & rate limiting**: LiteLLM handles cost tracking, rate limits, and fallbacks at the proxy layer
+
+## Environment Variables
+
+```bash
+# Required for providers you want to use (set in LiteLLM config or env)
+export OPENAI_API_KEY="sk-..."
+export ANTHROPIC_API_KEY="sk-ant-..."
+
+# Optional: point to a non-default LiteLLM proxy
+export LITELLM_PROXY_URL="http://localhost:4000"
+```
+
+## Troubleshooting
+
+**Connection errors?**
+- Make sure LiteLLM is running: `litellm --model gpt-4o`
+- Check the URL is correct (default: `http://localhost:4000`)
+
+**Model not found?**
+- Verify the model name matches what's configured in LiteLLM
+- Run `curl http://localhost:4000/models` to see available models
+
+**Tool calls not working?**
+- Ensure the underlying model supports tool use (GPT-4o, Claude, Gemini)
+- Check LiteLLM logs for any provider-specific errors
+
+## Learn More
+
+- [gollem GitHub](https://github.com/fugue-labs/gollem)
+- [gollem API Reference](https://pkg.go.dev/github.com/fugue-labs/gollem/core)
+- [LiteLLM Proxy Docs](https://docs.litellm.ai/docs/simple_proxy)
+- [LiteLLM Supported Models](https://docs.litellm.ai/docs/providers)
diff --git a/cookbook/gollem_go_agent_framework/basic/main.go b/cookbook/gollem_go_agent_framework/basic/main.go
new file mode 100644
index 00000000..838149a8
--- /dev/null
+++ b/cookbook/gollem_go_agent_framework/basic/main.go
@@ -0,0 +1,41 @@
+// Basic gollem agent connected to a LiteLLM proxy.
+//
+// Usage:
+//
+// litellm --model gpt-4o # start proxy in another terminal
+// go run ./basic
+package main
+
+import (
+ "context"
+ "fmt"
+ "log"
+ "os"
+
+ "github.com/fugue-labs/gollem/core"
+ "github.com/fugue-labs/gollem/provider/openai"
+)
+
+func main() {
+ proxyURL := "http://localhost:4000"
+ if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
+ proxyURL = u
+ }
+
+ // Connect to LiteLLM proxy. NewLiteLLM creates an OpenAI-compatible
+ // provider pointed at the given URL.
+ model := openai.NewLiteLLM(proxyURL,
+ openai.WithModel("gpt-4o"), // any model name configured in LiteLLM
+ )
+
+ // Create and run a simple agent.
+ agent := core.NewAgent[string](model,
+ core.WithSystemPrompt[string]("You are a helpful assistant. Be concise."),
+ )
+
+ result, err := agent.Run(context.Background(), "Explain quantum computing in two sentences.")
+ if err != nil {
+ log.Fatal(err)
+ }
+ fmt.Println(result.Output)
+}
diff --git a/cookbook/gollem_go_agent_framework/go.mod b/cookbook/gollem_go_agent_framework/go.mod
new file mode 100644
index 00000000..a8dc9365
--- /dev/null
+++ b/cookbook/gollem_go_agent_framework/go.mod
@@ -0,0 +1,5 @@
+module github.com/BerriAI/litellm/cookbook/gollem_go_agent_framework
+
+go 1.26.3
+
+require github.com/fugue-labs/gollem v0.1.0
diff --git a/cookbook/gollem_go_agent_framework/go.sum b/cookbook/gollem_go_agent_framework/go.sum
new file mode 100644
index 00000000..1eb6c5ac
--- /dev/null
+++ b/cookbook/gollem_go_agent_framework/go.sum
@@ -0,0 +1,2 @@
+github.com/fugue-labs/gollem v0.1.0 h1:QexYnvkb44QZFEljgAePqMIGZjgsbk0Y5GJ2jYYgfa8=
+github.com/fugue-labs/gollem v0.1.0/go.mod h1:htW1YO81uysSKVOkYJtxhGCFrzm+36HBFxEWuECoHKQ=
diff --git a/cookbook/gollem_go_agent_framework/proxy_config.yaml b/cookbook/gollem_go_agent_framework/proxy_config.yaml
new file mode 100644
index 00000000..18265a00
--- /dev/null
+++ b/cookbook/gollem_go_agent_framework/proxy_config.yaml
@@ -0,0 +1,16 @@
+model_list:
+ - model_name: gpt-4o
+ litellm_params:
+ model: openai/gpt-4o
+ api_key: os.environ/OPENAI_API_KEY
+
+ - model_name: claude-sonnet
+ litellm_params:
+ model: anthropic/claude-sonnet-4-20250514
+ api_key: os.environ/ANTHROPIC_API_KEY
+
+ - model_name: gemini-pro
+ litellm_params:
+ model: vertex_ai/gemini-2.0-flash
+ vertex_project: my-project
+ vertex_location: us-central1
diff --git a/cookbook/gollem_go_agent_framework/streaming/main.go b/cookbook/gollem_go_agent_framework/streaming/main.go
new file mode 100644
index 00000000..42bc9bbe
--- /dev/null
+++ b/cookbook/gollem_go_agent_framework/streaming/main.go
@@ -0,0 +1,56 @@
+// Streaming responses from gollem through LiteLLM.
+//
+// Uses Go 1.23+ range-over-function iterators for real-time token
+// streaming via LiteLLM's SSE passthrough.
+//
+// Usage:
+//
+// litellm --model gpt-4o
+// go run ./streaming
+package main
+
+import (
+ "context"
+ "fmt"
+ "log"
+ "os"
+
+ "github.com/fugue-labs/gollem/core"
+ "github.com/fugue-labs/gollem/provider/openai"
+)
+
+func main() {
+ proxyURL := "http://localhost:4000"
+ if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
+ proxyURL = u
+ }
+
+ model := openai.NewLiteLLM(proxyURL,
+ openai.WithModel("gpt-4o"),
+ )
+
+ agent := core.NewAgent[string](model)
+
+ // RunStream returns a streaming result that yields tokens as they arrive.
+ stream, err := agent.RunStream(context.Background(), "Write a haiku about distributed systems")
+ if err != nil {
+ log.Fatal(err)
+ }
+
+ // StreamText yields text chunks in real-time.
+ // The boolean argument controls whether deltas (true) or accumulated
+ // text (false) is returned.
+ fmt.Print("Response: ")
+ for text, err := range stream.StreamText(true) {
+ if err != nil {
+ log.Fatal(err)
+ }
+ fmt.Print(text)
+ }
+ fmt.Println()
+
+ // After streaming completes, the final response is available.
+ resp := stream.Response()
+ fmt.Printf("\nTokens used: input=%d, output=%d\n",
+ resp.Usage.InputTokens, resp.Usage.OutputTokens)
+}
diff --git a/cookbook/gollem_go_agent_framework/tools/main.go b/cookbook/gollem_go_agent_framework/tools/main.go
new file mode 100644
index 00000000..ed41a95f
--- /dev/null
+++ b/cookbook/gollem_go_agent_framework/tools/main.go
@@ -0,0 +1,64 @@
+// Gollem agent with type-safe tools through LiteLLM.
+//
+// The tool parameters are Go structs ā gollem generates the JSON schema
+// automatically at compile time. LiteLLM passes tool definitions through
+// transparently to the underlying provider.
+//
+// Usage:
+//
+// litellm --model gpt-4o
+// go run ./tools
+package main
+
+import (
+ "context"
+ "fmt"
+ "log"
+ "os"
+
+ "github.com/fugue-labs/gollem/core"
+ "github.com/fugue-labs/gollem/provider/openai"
+)
+
+// WeatherParams defines the tool's input schema via struct tags.
+// The JSON schema is generated at compile time ā no runtime reflection needed.
+type WeatherParams struct {
+ City string `json:"city" description:"City name to get weather for"`
+ Unit string `json:"unit,omitempty" description:"Temperature unit: celsius or fahrenheit"`
+}
+
+func main() {
+ proxyURL := "http://localhost:4000"
+ if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
+ proxyURL = u
+ }
+
+ model := openai.NewLiteLLM(proxyURL,
+ openai.WithModel("gpt-4o"),
+ )
+
+ // Define a type-safe tool. The function signature enforces correct types.
+ weatherTool := core.FuncTool[WeatherParams](
+ "get_weather",
+ "Get current weather for a city",
+ func(ctx context.Context, p WeatherParams) (string, error) {
+ unit := p.Unit
+ if unit == "" {
+ unit = "fahrenheit"
+ }
+ // In production, call a real weather API here.
+ return fmt.Sprintf("Weather in %s: 72Ā°F (22Ā°C), sunny", p.City), nil
+ },
+ )
+
+ agent := core.NewAgent[string](model,
+ core.WithTools[string](weatherTool),
+ core.WithSystemPrompt[string]("You are a helpful weather assistant. Use the get_weather tool to answer weather questions."),
+ )
+
+ result, err := agent.Run(context.Background(), "What's the weather like in San Francisco and Tokyo?")
+ if err != nil {
+ log.Fatal(err)
+ }
+ fmt.Println(result.Output)
+}
diff --git a/cookbook/google_adk_litellm_tutorial.ipynb b/cookbook/google_adk_litellm_tutorial.ipynb
new file mode 100644
index 00000000..27914edb
--- /dev/null
+++ b/cookbook/google_adk_litellm_tutorial.ipynb
@@ -0,0 +1,412 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "7aa8875d",
+ "metadata": {},
+ "source": [
+ "# Google ADK with LiteLLM\n",
+ "\n",
+ "Use Google ADK with LiteLLM Python SDK, LiteLLM Proxy.\n",
+ "\n",
+ "This tutorial shows you how to create intelligent agents using Agent Development Kit (ADK) with support for multiple Large Language Model (LLM) providers through LiteLLM."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a4d249c3",
+ "metadata": {},
+ "source": [
+ "## Overview\n",
+ "\n",
+ "ADK (Agent Development Kit) allows you to build intelligent agents powered by LLMs. By integrating with LiteLLM, you can:\n",
+ "\n",
+ "- Use multiple LLM providers (OpenAI, Anthropic, Google, etc.)\n",
+ "- Switch easily between models from different providers\n",
+ "- Connect to a LiteLLM proxy for centralized model management"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a0bbb56b",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "\n",
+ "- Python environment setup\n",
+ "- API keys for model providers (OpenAI, Anthropic, Google AI Studio)\n",
+ "- Basic understanding of LLMs and agent concepts"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7fee50a8",
+ "metadata": {},
+ "source": [
+ "## Installation"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "44106a23",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Install dependencies\n",
+ "!pip install google-adk litellm"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2171740a",
+ "metadata": {},
+ "source": [
+ "## 1. Setting Up Environment"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6695807e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Setup environment and API keys\n",
+ "import os\n",
+ "import asyncio\n",
+ "from google.adk.agents import Agent\n",
+ "from google.adk.models.lite_llm import LiteLlm # For multi-model support\n",
+ "from google.adk.sessions import InMemorySessionService\n",
+ "from google.adk.runners import Runner\n",
+ "from google.genai import types\n",
+ "import litellm # Import for proxy configuration\n",
+ "\n",
+ "# Set your API keys\n",
+ "os.environ['GOOGLE_API_KEY'] = 'your-google-api-key' # For Gemini models\n",
+ "os.environ['OPENAI_API_KEY'] = 'your-openai-api-key' # For OpenAI models\n",
+ "os.environ['ANTHROPIC_API_KEY'] = 'your-anthropic-api-key' # For Claude models\n",
+ "\n",
+ "# Define model constants for cleaner code\n",
+ "MODEL_GEMINI_PRO = 'gemini-1.5-pro'\n",
+ "MODEL_GPT_4O = 'openai/gpt-4o'\n",
+ "MODEL_CLAUDE_SONNET = 'anthropic/claude-3-sonnet-20240229'"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d2b1ed59",
+ "metadata": {},
+ "source": [
+ "## 2. Define a Simple Tool"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "04b3ef5b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Weather tool implementation\n",
+ "def get_weather(city: str) -> dict:\n",
+ " \"\"\"Retrieves the current weather report for a specified city.\"\"\"\n",
+ " print(f'Tool: get_weather called for city: {city}')\n",
+ "\n",
+ " # Mock weather data\n",
+ " mock_weather_db = {\n",
+ " 'newyork': {\n",
+ " 'status': 'success',\n",
+ " 'report': 'The weather in New York is sunny with a temperature of 25Ā°C.'\n",
+ " },\n",
+ " 'london': {\n",
+ " 'status': 'success',\n",
+ " 'report': \"It's cloudy in London with a temperature of 15Ā°C.\"\n",
+ " },\n",
+ " 'tokyo': {\n",
+ " 'status': 'success',\n",
+ " 'report': 'Tokyo is experiencing light rain and a temperature of 18Ā°C.'\n",
+ " },\n",
+ " }\n",
+ "\n",
+ " city_normalized = city.lower().replace(' ', '')\n",
+ "\n",
+ " if city_normalized in mock_weather_db:\n",
+ " return mock_weather_db[city_normalized]\n",
+ " else:\n",
+ " return {\n",
+ " 'status': 'error',\n",
+ " 'error_message': f\"Sorry, I don't have weather information for '{city}'.\"\n",
+ " }"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "727b15c9",
+ "metadata": {},
+ "source": [
+ "## 3. Helper Function for Agent Interaction"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f77449bf",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Agent interaction helper function\n",
+ "async def call_agent_async(query: str, runner, user_id, session_id):\n",
+ " \"\"\"Sends a query to the agent and prints the final response.\"\"\"\n",
+ " print(f'\\n>>> User Query: {query}')\n",
+ "\n",
+ " content = types.Content(role='user', parts=[types.Part(text=query)])\n",
+ " final_response_text = 'Agent did not produce a final response.'\n",
+ "\n",
+ " async for event in runner.run_async(\n",
+ " user_id=user_id,\n",
+ " session_id=session_id,\n",
+ " new_message=content\n",
+ " ):\n",
+ " if event.is_final_response():\n",
+ " if event.content and event.content.parts:\n",
+ " final_response_text = event.content.parts[0].text\n",
+ " break\n",
+ " print(f'<<< Agent Response: {final_response_text}')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0ac87987",
+ "metadata": {},
+ "source": [
+ "## 4. Using Different Model Providers with ADK\n",
+ "\n",
+ "### 4.1 Using OpenAI Models"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e167d557",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# OpenAI model implementation\n",
+ "weather_agent_gpt = Agent(\n",
+ " name='weather_agent_gpt',\n",
+ " model=LiteLlm(model=MODEL_GPT_4O),\n",
+ " description='Provides weather information using OpenAI\\'s GPT.',\n",
+ " instruction=(\n",
+ " 'You are a helpful weather assistant powered by GPT-4o. '\n",
+ " \"Use the 'get_weather' tool for city weather requests. \"\n",
+ " 'Present information clearly.'\n",
+ " ),\n",
+ " tools=[get_weather],\n",
+ ")\n",
+ "\n",
+ "session_service_gpt = InMemorySessionService()\n",
+ "session_gpt = session_service_gpt.create_session(\n",
+ " app_name='weather_app', user_id='user_1', session_id='session_gpt'\n",
+ ")\n",
+ "\n",
+ "runner_gpt = Runner(\n",
+ " agent=weather_agent_gpt,\n",
+ " app_name='weather_app',\n",
+ " session_service=session_service_gpt,\n",
+ ")\n",
+ "\n",
+ "async def test_gpt_agent():\n",
+ " print('\\n--- Testing GPT Agent ---')\n",
+ " await call_agent_async(\n",
+ " \"What's the weather in London?\",\n",
+ " runner=runner_gpt,\n",
+ " user_id='user_1',\n",
+ " session_id='session_gpt',\n",
+ " )\n",
+ "\n",
+ "# To execute in a notebook cell:\n",
+ "# await test_gpt_agent()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f9cb0613",
+ "metadata": {},
+ "source": [
+ "### 4.2 Using Anthropic Models"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "1c653665",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Anthropic model implementation\n",
+ "weather_agent_claude = Agent(\n",
+ " name='weather_agent_claude',\n",
+ " model=LiteLlm(model=MODEL_CLAUDE_SONNET),\n",
+ " description='Provides weather information using Anthropic\\'s Claude.',\n",
+ " instruction=(\n",
+ " 'You are a helpful weather assistant powered by Claude Sonnet. '\n",
+ " \"Use the 'get_weather' tool for city weather requests. \"\n",
+ " 'Present information clearly.'\n",
+ " ),\n",
+ " tools=[get_weather],\n",
+ ")\n",
+ "\n",
+ "session_service_claude = InMemorySessionService()\n",
+ "session_claude = session_service_claude.create_session(\n",
+ " app_name='weather_app', user_id='user_1', session_id='session_claude'\n",
+ ")\n",
+ "\n",
+ "runner_claude = Runner(\n",
+ " agent=weather_agent_claude,\n",
+ " app_name='weather_app',\n",
+ " session_service=session_service_claude,\n",
+ ")\n",
+ "\n",
+ "async def test_claude_agent():\n",
+ " print('\\n--- Testing Claude Agent ---')\n",
+ " await call_agent_async(\n",
+ " \"What's the weather in Tokyo?\",\n",
+ " runner=runner_claude,\n",
+ " user_id='user_1',\n",
+ " session_id='session_claude',\n",
+ " )\n",
+ "\n",
+ "# To execute in a notebook cell:\n",
+ "# await test_claude_agent()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bf9d863b",
+ "metadata": {},
+ "source": [
+ "### 4.3 Using Google's Gemini Models"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "83f49d0a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Gemini model implementation\n",
+ "weather_agent_gemini = Agent(\n",
+ " name='weather_agent_gemini',\n",
+ " model=MODEL_GEMINI_PRO,\n",
+ " description='Provides weather information using Google\\'s Gemini.',\n",
+ " instruction=(\n",
+ " 'You are a helpful weather assistant powered by Gemini Pro. '\n",
+ " \"Use the 'get_weather' tool for city weather requests. \"\n",
+ " 'Present information clearly.'\n",
+ " ),\n",
+ " tools=[get_weather],\n",
+ ")\n",
+ "\n",
+ "session_service_gemini = InMemorySessionService()\n",
+ "session_gemini = session_service_gemini.create_session(\n",
+ " app_name='weather_app', user_id='user_1', session_id='session_gemini'\n",
+ ")\n",
+ "\n",
+ "runner_gemini = Runner(\n",
+ " agent=weather_agent_gemini,\n",
+ " app_name='weather_app',\n",
+ " session_service=session_service_gemini,\n",
+ ")\n",
+ "\n",
+ "async def test_gemini_agent():\n",
+ " print('\\n--- Testing Gemini Agent ---')\n",
+ " await call_agent_async(\n",
+ " \"What's the weather in New York?\",\n",
+ " runner=runner_gemini,\n",
+ " user_id='user_1',\n",
+ " session_id='session_gemini',\n",
+ " )\n",
+ "\n",
+ "# To execute in a notebook cell:\n",
+ "# await test_gemini_agent()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "93bc5fd0",
+ "metadata": {},
+ "source": [
+ "## 5. Using LiteLLM Proxy with ADK"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b4275151",
+ "metadata": {},
+ "source": [
+ "| Variable | Description |\n",
+ "|----------|-------------|\n",
+ "| `LITELLM_PROXY_API_KEY` | The API key for the LiteLLM proxy |\n",
+ "| `LITELLM_PROXY_API_BASE` | The base URL for the LiteLLM proxy |\n",
+ "| `USE_LITELLM_PROXY` or `litellm.use_litellm_proxy` | When set to True, your request will be sent to LiteLLM proxy. |"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "256530a6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# LiteLLM proxy integration\n",
+ "os.environ['LITELLM_PROXY_API_KEY'] = 'your-litellm-proxy-api-key'\n",
+ "os.environ['LITELLM_PROXY_API_BASE'] = 'your-litellm-proxy-url' # e.g., 'http://localhost:4000'\n",
+ "litellm.use_litellm_proxy = True\n",
+ "\n",
+ "weather_agent_proxy_env = Agent(\n",
+ " name='weather_agent_proxy_env',\n",
+ " model=LiteLlm(model='gpt-4o'),\n",
+ " description='Provides weather information using a model from LiteLLM proxy.',\n",
+ " instruction=(\n",
+ " 'You are a helpful weather assistant. '\n",
+ " \"Use the 'get_weather' tool for city weather requests. \"\n",
+ " 'Present information clearly.'\n",
+ " ),\n",
+ " tools=[get_weather],\n",
+ ")\n",
+ "\n",
+ "session_service_proxy_env = InMemorySessionService()\n",
+ "session_proxy_env = session_service_proxy_env.create_session(\n",
+ " app_name='weather_app', user_id='user_1', session_id='session_proxy_env'\n",
+ ")\n",
+ "\n",
+ "runner_proxy_env = Runner(\n",
+ " agent=weather_agent_proxy_env,\n",
+ " app_name='weather_app',\n",
+ " session_service=session_service_proxy_env,\n",
+ ")\n",
+ "\n",
+ "async def test_proxy_env_agent():\n",
+ " print('\\n--- Testing Proxy-enabled Agent (Environment Variables) ---')\n",
+ " await call_agent_async(\n",
+ " \"What's the weather in London?\",\n",
+ " runner=runner_proxy_env,\n",
+ " user_id='user_1',\n",
+ " session_id='session_proxy_env',\n",
+ " )\n",
+ "\n",
+ "# To execute in a notebook cell:\n",
+ "# await test_proxy_env_agent()"
+ ]
+ }
+ ],
+ "metadata": {
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/cookbook/liteLLM_A121_Jurrasic_example.ipynb b/cookbook/liteLLM_A121_Jurrasic_example.ipynb
new file mode 100644
index 00000000..f975b97e
--- /dev/null
+++ b/cookbook/liteLLM_A121_Jurrasic_example.ipynb
@@ -0,0 +1,251 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "provenance": []
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# LiteLLM A121 Tutorial\n",
+ "\n",
+ "This walks through using A121 Jurassic models\n",
+ "* j2-light\n",
+ "* j2-mid\n",
+ "* j2-ultra"
+ ],
+ "metadata": {
+ "id": "LeFYo8iqcn5g"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "GslPQFmaZsp-"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install litellm"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from litellm import completion\n",
+ "import os"
+ ],
+ "metadata": {
+ "id": "P3cKiqURZx7P"
+ },
+ "execution_count": 2,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Set A121 Keys\n",
+ "You can get a free key from https://studio.ai21.com/account/api-key"
+ ],
+ "metadata": {
+ "id": "tmTvA1_GaNU4"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "os.environ[\"AI21_API_KEY\"] = \"\""
+ ],
+ "metadata": {
+ "id": "_xX8LmxAZ2vp"
+ },
+ "execution_count": 5,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# A121 Supported Models:\n",
+ "https://studio.ai21.com/foundation-models"
+ ],
+ "metadata": {
+ "id": "Fx5ZfJTLbF0A"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## J2-light Call"
+ ],
+ "metadata": {
+ "id": "H0tl-0Z3bDaL"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "messages = [{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}]\n",
+ "response = completion(model=\"j2-light\", messages=messages)\n",
+ "response"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "DZnApsJUZ_I2",
+ "outputId": "b5707cbe-f67c-47f7-bac5-a7b8af1ba815"
+ },
+ "execution_count": 6,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "
+
+
+
+
+
+### Pre-Requisites
+- Setup LiteLLM Proxy Prometheus Metrics https://docs.litellm.ai/docs/proxy/prometheus
diff --git a/cookbook/litellm_proxy_server/mcp/mcp_with_litellm_proxy.py b/cookbook/litellm_proxy_server/mcp/mcp_with_litellm_proxy.py
new file mode 100644
index 00000000..cc933027
--- /dev/null
+++ b/cookbook/litellm_proxy_server/mcp/mcp_with_litellm_proxy.py
@@ -0,0 +1,37 @@
+"""
+Use LiteLLM Proxy MCP Gateway to call MCP tools.
+
+When using LiteLLM Proxy, you can use the same MCP tools across all your LLM providers.
+"""
+
+import openai
+
+client = openai.OpenAI(
+ api_key="sk-1234", # paste your litellm proxy api key here
+ base_url="http://localhost:4000", # paste your litellm proxy base url here
+)
+print("Making API request to Responses API with MCP tools")
+
+response = client.responses.create(
+ model="gpt-5",
+ input=[
+ {
+ "role": "user",
+ "content": "give me TLDR of what BerriAI/litellm repo is about",
+ "type": "message",
+ }
+ ],
+ tools=[
+ {
+ "type": "mcp",
+ "server_label": "litellm",
+ "server_url": "litellm_proxy",
+ "require_approval": "never",
+ }
+ ],
+ stream=True,
+ tool_choice="required",
+)
+
+for chunk in response:
+ print("response chunk: ", chunk)
diff --git a/cookbook/litellm_proxy_server/readme.md b/cookbook/litellm_proxy_server/readme.md
new file mode 100644
index 00000000..2c1eab72
--- /dev/null
+++ b/cookbook/litellm_proxy_server/readme.md
@@ -0,0 +1,177 @@
+# liteLLM Proxy Server: 50+ LLM Models, Error Handling, Caching
+
+### Azure, Llama2, OpenAI, Claude, Hugging Face, Replicate Models
+
+[](https://pypi.org/project/litellm/)
+[](https://pypi.org/project/litellm/0.1.1/)
+
+[](https://github.com/BerriAI/litellm)
+
+[](https://railway.app/template/DYqQAW?referralCode=t3ukrU)
+
+
+
+## What does liteLLM proxy do
+
+- Make `/chat/completions` requests for 50+ LLM models **Azure, OpenAI, Replicate, Anthropic, Hugging Face**
+
+ Example: for `model` use `claude-2`, `gpt-3.5`, `gpt-4`, `command-nightly`, `stabilityai/stablecode-completion-alpha-3b-4k`
+
+ ```json
+ {
+ "model": "replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1",
+ "messages": [
+ {
+ "content": "Hello, whats the weather in San Francisco??",
+ "role": "user"
+ }
+ ]
+ }
+ ```
+
+- **Consistent Input/Output** Format
+ - Call all models using the OpenAI format - `completion(model, messages)`
+ - Text responses will always be available at `['choices'][0]['message']['content']`
+- **Error Handling** Using Model Fallbacks (if `GPT-4` fails, try `llama2`)
+- **Logging** - Log Requests, Responses and Errors to `Supabase`, `Posthog`, `Mixpanel`, `Sentry`, `Lunary`,`Athina`, `Helicone` (Any of the supported providers here: https://litellm.readthedocs.io/en/latest/advanced/
+
+ **Example: Logs sent to Supabase**
+ 
+
+- **Token Usage & Spend** - Track Input + Completion tokens used + Spend/model
+- **Caching** - Implementation of Semantic Caching
+- **Streaming & Async Support** - Return generators to stream text responses
+
+## API Endpoints
+
+### `/chat/completions` (POST)
+
+This endpoint is used to generate chat completions for 50+ support LLM API Models. Use llama2, GPT-4, Claude2 etc
+
+#### Input
+
+This API endpoint accepts all inputs in raw JSON and expects the following inputs
+
+- `model` (string, required): ID of the model to use for chat completions. See all supported models [here]: (https://litellm.readthedocs.io/en/latest/supported/):
+ eg `gpt-3.5-turbo`, `gpt-4`, `claude-2`, `command-nightly`, `stabilityai/stablecode-completion-alpha-3b-4k`
+- `messages` (array, required): A list of messages representing the conversation context. Each message should have a `role` (system, user, assistant, or function), `content` (message text), and `name` (for function role).
+- Additional Optional parameters: `temperature`, `functions`, `function_call`, `top_p`, `n`, `stream`. See the full list of supported inputs here: https://litellm.readthedocs.io/en/latest/input/
+
+#### Example JSON body
+
+For claude-2
+
+```json
+{
+ "model": "claude-2",
+ "messages": [
+ {
+ "content": "Hello, whats the weather in San Francisco??",
+ "role": "user"
+ }
+ ]
+}
+```
+
+### Making an API request to the Proxy Server
+
+```python
+import requests
+import json
+
+# TODO: use your URL
+url = "http://localhost:5000/chat/completions"
+
+payload = json.dumps({
+ "model": "gpt-3.5-turbo",
+ "messages": [
+ {
+ "content": "Hello, whats the weather in San Francisco??",
+ "role": "user"
+ }
+ ]
+})
+headers = {
+ 'Content-Type': 'application/json'
+}
+response = requests.request("POST", url, headers=headers, data=payload)
+print(response.text)
+
+```
+
+### Output [Response Format]
+
+Responses from the server are given in the following format.
+All responses from the server are returned in the following format (for all LLM models). More info on output here: https://litellm.readthedocs.io/en/latest/output/
+
+```json
+{
+ "choices": [
+ {
+ "finish_reason": "stop",
+ "index": 0,
+ "message": {
+ "content": "I'm sorry, but I don't have the capability to provide real-time weather information. However, you can easily check the weather in San Francisco by searching online or using a weather app on your phone.",
+ "role": "assistant"
+ }
+ }
+ ],
+ "created": 1691790381,
+ "id": "chatcmpl-7mUFZlOEgdohHRDx2UpYPRTejirzb",
+ "model": "gpt-3.5-turbo-0613",
+ "object": "chat.completion",
+ "usage": {
+ "completion_tokens": 41,
+ "prompt_tokens": 16,
+ "total_tokens": 57
+ }
+}
+```
+
+## Installation & Usage
+
+### Running Locally
+
+1. Clone liteLLM repository to your local machine:
+ ```
+ git clone https://github.com/BerriAI/liteLLM-proxy
+ ```
+2. Install the required dependencies using pip
+ ```
+ pip install -r requirements.txt
+ ```
+3. Set your LLM API keys
+ ```
+ os.environ['OPENAI_API_KEY]` = "YOUR_API_KEY"
+ or
+ set OPENAI_API_KEY in your .env file
+ ```
+4. Run the server:
+ ```
+ python main.py
+ ```
+
+## Deploying
+
+1. Quick Start: Deploy on Railway
+
+ [](https://railway.app/template/DYqQAW?referralCode=t3ukrU)
+
+2. `GCP`, `AWS`, `Azure`
+ This project includes a `Dockerfile` allowing you to build and deploy a Docker Project on your providers
+
+# Support / Talk with founders
+
+- [Our calendar š](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
+- [Community Discord š](https://discord.gg/wuPM9dRgDw)
+- Our emails āļø ishaan@berri.ai / krrish@berri.ai
+
+## Roadmap
+
+- [ ] Support hosted db (e.g. Supabase)
+- [ ] Easily send data to places like posthog and sentry.
+- [ ] Add a hot-cache for project spend logs - enables fast checks for user + project limitings
+- [ ] Implement user-based rate-limiting
+- [ ] Spending controls per project - expose key creation endpoint
+- [ ] Need to store a keys db -> mapping created keys to their alias (i.e. project name)
+- [ ] Easily add new models as backups / as the entry-point (add this to the available model list)
diff --git a/cookbook/litellm_proxy_server/secret_manager/custom_secret_manager_config.yaml b/cookbook/litellm_proxy_server/secret_manager/custom_secret_manager_config.yaml
new file mode 100644
index 00000000..3598a9b1
--- /dev/null
+++ b/cookbook/litellm_proxy_server/secret_manager/custom_secret_manager_config.yaml
@@ -0,0 +1,20 @@
+general_settings:
+ master_key: os.environ/LITELLM_MASTER_KEY
+ key_management_system: "custom"
+ key_management_settings:
+ custom_secret_manager: my_secret_manager.InMemorySecretManager
+ store_virtual_keys: true
+ prefix_for_stored_virtual_keys: "litellm/"
+ access_mode: "read_and_write"
+
+model_list:
+ - model_name: gpt-4
+ litellm_params:
+ model: openai/gpt-4
+ api_key: os.environ/OPENAI_API_KEY # Read from custom secret manager
+
+ - model_name: claude-3-5-sonnet
+ litellm_params:
+ model: anthropic/claude-3-5-sonnet-20241022
+ api_key: os.environ/ANTHROPIC_API_KEY # Read from custom secret manager
+
diff --git a/cookbook/litellm_proxy_server/secret_manager/my_secret_manager.py b/cookbook/litellm_proxy_server/secret_manager/my_secret_manager.py
new file mode 100644
index 00000000..65c7f754
--- /dev/null
+++ b/cookbook/litellm_proxy_server/secret_manager/my_secret_manager.py
@@ -0,0 +1,80 @@
+"""
+Example custom secret manager for LiteLLM Proxy.
+
+This is a simple in-memory secret manager for testing purposes.
+In production, replace this with your actual secret management system.
+"""
+
+from typing import Optional, Union
+
+import httpx
+
+from litellm.integrations.custom_secret_manager import CustomSecretManager
+
+
+class InMemorySecretManager(CustomSecretManager):
+ def __init__(self):
+ super().__init__(secret_manager_name="in_memory_secrets")
+ # Store your secrets in memory
+ print("INITIALIZING CUSTOM SECRET MANAGER IN MEMORY")
+ self.secrets = {}
+ print("CUSTOM SECRET MANAGER IN MEMORY INITIALIZED")
+
+ async def async_read_secret(
+ self,
+ secret_name: str,
+ optional_params: Optional[dict] = None,
+ timeout: Optional[Union[float, httpx.Timeout]] = None,
+ ) -> Optional[str]:
+ """Read secret asynchronously"""
+ print("READING SECRET ASYNCHRONOUSLY")
+ print("SECRET NAME: %s", secret_name)
+ print("SECRET: %s", self.secrets.get(secret_name))
+ return self.secrets.get(secret_name)
+
+ def sync_read_secret(
+ self,
+ secret_name: str,
+ optional_params: Optional[dict] = None,
+ timeout: Optional[Union[float, httpx.Timeout]] = None,
+ ) -> Optional[str]:
+ """Read secret synchronously"""
+ from litellm._logging import verbose_proxy_logger
+
+ verbose_proxy_logger.info(
+ f"CUSTOM SECRET MANAGER: LOOKING FOR SECRET: {secret_name}"
+ )
+ value = self.secrets.get(secret_name)
+ verbose_proxy_logger.info(f"CUSTOM SECRET MANAGER: READ SECRET: {value}")
+ return value
+
+ async def async_write_secret(
+ self,
+ secret_name: str,
+ secret_value: str,
+ description: Optional[str] = None,
+ optional_params: Optional[dict] = None,
+ timeout: Optional[Union[float, httpx.Timeout]] = None,
+ tags: Optional[Union[dict, list]] = None,
+ ) -> dict:
+ """Write a secret to the in-memory store"""
+ self.secrets[secret_name] = secret_value
+ print("ALL SECRETS=%s", self.secrets)
+ return {
+ "status": "success",
+ "secret_name": secret_name,
+ "description": description,
+ }
+
+ async def async_delete_secret(
+ self,
+ secret_name: str,
+ recovery_window_in_days: Optional[int] = 7,
+ optional_params: Optional[dict] = None,
+ timeout: Optional[Union[float, httpx.Timeout]] = None,
+ ) -> dict:
+ """Delete a secret from the in-memory store"""
+ if secret_name in self.secrets:
+ del self.secrets[secret_name]
+ return {"status": "deleted", "secret_name": secret_name}
+ return {"status": "not_found", "secret_name": secret_name}
diff --git a/cookbook/litellm_router/error_log.txt b/cookbook/litellm_router/error_log.txt
new file mode 100644
index 00000000..983b47cb
--- /dev/null
+++ b/cookbook/litellm_router/error_log.txt
@@ -0,0 +1,1004 @@
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: Expecting value: line 1 column 1 (char 0)
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Exception: 'Response' object has no attribute 'get'
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Exception: 'Response' object has no attribute 'get'
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Exception: 'Response' object has no attribute 'get'
+
diff --git a/cookbook/litellm_router/load_test_proxy.py b/cookbook/litellm_router/load_test_proxy.py
new file mode 100644
index 00000000..9ae6e764
--- /dev/null
+++ b/cookbook/litellm_router/load_test_proxy.py
@@ -0,0 +1,148 @@
+import sys
+import os
+from dotenv import load_dotenv
+
+load_dotenv()
+
+sys.path.insert(
+ 0, os.path.abspath("../..")
+) # Adds the parent directory to the system path
+
+from litellm import Router
+import litellm
+
+litellm.set_verbose = False
+os.environ.pop("AZURE_AD_TOKEN")
+
+model_list = [
+ { # list of model deployments
+ "model_name": "gpt-3.5-turbo", # model alias
+ "litellm_params": { # params for litellm completion/embedding call
+ "model": "azure/chatgpt-v-2", # actual model name
+ "api_key": os.getenv("AZURE_API_KEY"),
+ "api_version": os.getenv("AZURE_API_VERSION"),
+ "api_base": os.getenv("AZURE_API_BASE"),
+ },
+ },
+ {
+ "model_name": "gpt-3.5-turbo",
+ "litellm_params": { # params for litellm completion/embedding call
+ "model": "azure/chatgpt-functioncalling",
+ "api_key": os.getenv("AZURE_API_KEY"),
+ "api_version": os.getenv("AZURE_API_VERSION"),
+ "api_base": os.getenv("AZURE_API_BASE"),
+ },
+ },
+ {
+ "model_name": "gpt-3.5-turbo",
+ "litellm_params": { # params for litellm completion/embedding call
+ "model": "gpt-3.5-turbo",
+ "api_key": os.getenv("OPENAI_API_KEY"),
+ },
+ },
+]
+router = Router(model_list=model_list)
+
+
+file_paths = [
+ "test_questions/question1.txt",
+ "test_questions/question2.txt",
+ "test_questions/question3.txt",
+]
+questions = []
+
+for file_path in file_paths:
+ try:
+ print(file_path)
+ with open(file_path, "r") as file:
+ content = file.read()
+ questions.append(content)
+ except FileNotFoundError as e:
+ print(f"File not found: {e}")
+ except Exception as e:
+ print(f"An error occurred: {e}")
+
+# for q in questions:
+# print(q)
+
+
+# make X concurrent calls to litellm.completion(model=gpt-35-turbo, messages=[]), pick a random question in questions array.
+# Allow me to tune X concurrent calls.. Log question, output/exception, response time somewhere
+# show me a summary of requests made, success full calls, failed calls. For failed calls show me the exceptions
+
+import concurrent.futures
+import random
+import time
+
+
+# Function to make concurrent calls to OpenAI API
+def make_openai_completion(question):
+ try:
+ start_time = time.time()
+ import openai
+
+ client = openai.OpenAI(
+ api_key=os.environ["OPENAI_API_KEY"], base_url="http://0.0.0.0:8000"
+ ) # base_url="http://0.0.0.0:8000",
+ response = client.chat.completions.create(
+ model="gpt-3.5-turbo",
+ messages=[
+ {
+ "role": "system",
+ "content": f"You are a helpful assistant. Answer this question{question}",
+ }
+ ],
+ )
+ print(response)
+ end_time = time.time()
+
+ # Log the request details
+ with open("request_log.txt", "a") as log_file:
+ log_file.write(
+ f"Question: {question[:100]}\nResponse ID:{response.id} Content:{response.choices[0].message.content[:10]}\nTime: {end_time - start_time:.2f} seconds\n\n"
+ )
+
+ return response
+ except Exception as e:
+ # Log exceptions for failed calls
+ with open("error_log.txt", "a") as error_log_file:
+ error_log_file.write(f"Question: {question[:100]}\nException: {str(e)}\n\n")
+ return None
+
+
+# Number of concurrent calls (you can adjust this)
+concurrent_calls = 100
+
+# List to store the futures of concurrent calls
+futures = []
+
+# Make concurrent calls
+with concurrent.futures.ThreadPoolExecutor(max_workers=concurrent_calls) as executor:
+ for _ in range(concurrent_calls):
+ random_question = random.choice(questions)
+ futures.append(executor.submit(make_openai_completion, random_question))
+
+# Wait for all futures to complete
+concurrent.futures.wait(futures)
+
+# Summarize the results
+successful_calls = 0
+failed_calls = 0
+
+for future in futures:
+ if future.result() is not None:
+ successful_calls += 1
+ else:
+ failed_calls += 1
+
+print("Load test Summary:")
+print(f"Total Requests: {concurrent_calls}")
+print(f"Successful Calls: {successful_calls}")
+print(f"Failed Calls: {failed_calls}")
+
+# Display content of the logs
+with open("request_log.txt", "r") as log_file:
+ print("\nRequest Log:\n", log_file.read())
+
+with open("error_log.txt", "r") as error_log_file:
+ print("\nError Log:\n", error_log_file.read())
diff --git a/cookbook/litellm_router/load_test_queuing.py b/cookbook/litellm_router/load_test_queuing.py
new file mode 100644
index 00000000..7d4d44b2
--- /dev/null
+++ b/cookbook/litellm_router/load_test_queuing.py
@@ -0,0 +1,164 @@
+import sys
+import os
+from dotenv import load_dotenv
+
+load_dotenv()
+
+sys.path.insert(
+ 0, os.path.abspath("../..")
+) # Adds the parent directory to the system path
+
+from litellm import Router
+import litellm
+
+litellm.set_verbose = False
+# os.environ.pop("AZURE_AD_TOKEN")
+
+model_list = [
+ { # list of model deployments
+ "model_name": "gpt-3.5-turbo", # model alias
+ "litellm_params": { # params for litellm completion/embedding call
+ "model": "azure/chatgpt-v-2", # actual model name
+ "api_key": os.getenv("AZURE_API_KEY"),
+ "api_version": os.getenv("AZURE_API_VERSION"),
+ "api_base": os.getenv("AZURE_API_BASE"),
+ },
+ },
+ {
+ "model_name": "gpt-3.5-turbo",
+ "litellm_params": { # params for litellm completion/embedding call
+ "model": "azure/chatgpt-functioncalling",
+ "api_key": os.getenv("AZURE_API_KEY"),
+ "api_version": os.getenv("AZURE_API_VERSION"),
+ "api_base": os.getenv("AZURE_API_BASE"),
+ },
+ },
+ {
+ "model_name": "gpt-3.5-turbo",
+ "litellm_params": { # params for litellm completion/embedding call
+ "model": "gpt-3.5-turbo",
+ "api_key": os.getenv("OPENAI_API_KEY"),
+ },
+ },
+]
+router = Router(model_list=model_list)
+
+
+file_paths = [
+ "test_questions/question1.txt",
+ "test_questions/question2.txt",
+ "test_questions/question3.txt",
+]
+questions = []
+
+for file_path in file_paths:
+ try:
+ print(file_path)
+ with open(file_path, "r") as file:
+ content = file.read()
+ questions.append(content)
+ except FileNotFoundError as e:
+ print(f"File not found: {e}")
+ except Exception as e:
+ print(f"An error occurred: {e}")
+
+# for q in questions:
+# print(q)
+
+
+# make X concurrent calls to litellm.completion(model=gpt-35-turbo, messages=[]), pick a random question in questions array.
+# Allow me to tune X concurrent calls.. Log question, output/exception, response time somewhere
+# show me a summary of requests made, success full calls, failed calls. For failed calls show me the exceptions
+
+import concurrent.futures
+import random
+import time
+
+
+# Function to make concurrent calls to OpenAI API
+def make_openai_completion(question):
+ try:
+ start_time = time.time()
+ import requests
+
+ data = {
+ "model": "gpt-3.5-turbo",
+ "messages": [
+ {
+ "role": "system",
+ "content": f"You are a helpful assistant. Answer this question{question}",
+ },
+ ],
+ }
+ response = requests.post("http://0.0.0.0:8000/queue/request", json=data)
+ response = response.json()
+ end_time = time.time()
+ # Log the request details
+ with open("request_log.txt", "a") as log_file:
+ log_file.write(
+ f"Question: {question[:100]}\nResponse ID: {response.get('id', 'N/A')} Url: {response.get('url', 'N/A')}\nTime: {end_time - start_time:.2f} seconds\n\n"
+ )
+
+ # polling the url
+ while True:
+ try:
+ url = response["url"]
+ polling_url = f"http://0.0.0.0:8000{url}"
+ polling_response = requests.get(polling_url)
+ polling_response = polling_response.json()
+ print("\n RESPONSE FROM POLLING JoB", polling_response)
+ status = polling_response["status"]
+ if status == "finished":
+ llm_response = polling_response["result"]
+ with open("response_log.txt", "a") as log_file:
+ log_file.write(
+ f"Response ID: {llm_response.get('id', 'NA')}\nLLM Response: {llm_response}\nTime: {end_time - start_time:.2f} seconds\n\n"
+ )
+
+ break
+ print(
+ f"POLLING JOB{polling_url}\nSTATUS: {status}, \n Response {polling_response}"
+ )
+ time.sleep(0.5)
+ except Exception as e:
+ print("got exception in polling", e)
+ break
+
+ return response
+ except Exception as e:
+ # Log exceptions for failed calls
+ with open("error_log.txt", "a") as error_log_file:
+ error_log_file.write(f"Question: {question[:100]}\nException: {str(e)}\n\n")
+ return None
+
+
+# Number of concurrent calls (you can adjust this)
+concurrent_calls = 10
+
+# List to store the futures of concurrent calls
+futures = []
+
+# Make concurrent calls
+with concurrent.futures.ThreadPoolExecutor(max_workers=concurrent_calls) as executor:
+ for _ in range(concurrent_calls):
+ random_question = random.choice(questions)
+ futures.append(executor.submit(make_openai_completion, random_question))
+
+# Wait for all futures to complete
+concurrent.futures.wait(futures)
+
+# Summarize the results
+successful_calls = 0
+failed_calls = 0
+
+for future in futures:
+ if future.done():
+ if future.result() is not None:
+ successful_calls += 1
+ else:
+ failed_calls += 1
+
+print("Load test Summary:")
+print(f"Total Requests: {concurrent_calls}")
+print(f"Successful Calls: {successful_calls}")
+print(f"Failed Calls: {failed_calls}")
diff --git a/cookbook/litellm_router/load_test_router.py b/cookbook/litellm_router/load_test_router.py
new file mode 100644
index 00000000..92533b6c
--- /dev/null
+++ b/cookbook/litellm_router/load_test_router.py
@@ -0,0 +1,143 @@
+import sys
+import os
+from dotenv import load_dotenv
+
+load_dotenv()
+
+sys.path.insert(
+ 0, os.path.abspath("../..")
+) # Adds the parent directory to the system path
+
+from litellm import Router
+import litellm
+
+litellm.set_verbose = False
+os.environ.pop("AZURE_AD_TOKEN")
+
+model_list = [
+ { # list of model deployments
+ "model_name": "gpt-3.5-turbo", # model alias
+ "litellm_params": { # params for litellm completion/embedding call
+ "model": "azure/chatgpt-v-2", # actual model name
+ "api_key": os.getenv("AZURE_API_KEY"),
+ "api_version": os.getenv("AZURE_API_VERSION"),
+ "api_base": os.getenv("AZURE_API_BASE"),
+ },
+ },
+ {
+ "model_name": "gpt-3.5-turbo",
+ "litellm_params": { # params for litellm completion/embedding call
+ "model": "azure/chatgpt-functioncalling",
+ "api_key": os.getenv("AZURE_API_KEY"),
+ "api_version": os.getenv("AZURE_API_VERSION"),
+ "api_base": os.getenv("AZURE_API_BASE"),
+ },
+ },
+ {
+ "model_name": "gpt-3.5-turbo",
+ "litellm_params": { # params for litellm completion/embedding call
+ "model": "gpt-3.5-turbo",
+ "api_key": os.getenv("OPENAI_API_KEY"),
+ },
+ },
+]
+router = Router(model_list=model_list)
+
+
+file_paths = [
+ "test_questions/question1.txt",
+ "test_questions/question2.txt",
+ "test_questions/question3.txt",
+]
+questions = []
+
+for file_path in file_paths:
+ try:
+ print(file_path)
+ with open(file_path, "r") as file:
+ content = file.read()
+ questions.append(content)
+ except FileNotFoundError as e:
+ print(f"File not found: {e}")
+ except Exception as e:
+ print(f"An error occurred: {e}")
+
+# for q in questions:
+# print(q)
+
+
+# make X concurrent calls to litellm.completion(model=gpt-35-turbo, messages=[]), pick a random question in questions array.
+# Allow me to tune X concurrent calls.. Log question, output/exception, response time somewhere
+# show me a summary of requests made, success full calls, failed calls. For failed calls show me the exceptions
+
+import concurrent.futures
+import random
+import time
+
+
+# Function to make concurrent calls to OpenAI API
+def make_openai_completion(question):
+ try:
+ start_time = time.time()
+ response = router.completion(
+ model="gpt-3.5-turbo",
+ messages=[
+ {
+ "role": "system",
+ "content": f"You are a helpful assistant. Answer this question{question}",
+ }
+ ],
+ )
+ print(response)
+ end_time = time.time()
+
+ # Log the request details
+ with open("request_log.txt", "a") as log_file:
+ log_file.write(
+ f"Question: {question[:100]}\nResponse: {response.choices[0].message.content}\nTime: {end_time - start_time:.2f} seconds\n\n"
+ )
+
+ return response
+ except Exception as e:
+ # Log exceptions for failed calls
+ with open("error_log.txt", "a") as error_log_file:
+ error_log_file.write(f"Question: {question[:100]}\nException: {str(e)}\n\n")
+ return None
+
+
+# Number of concurrent calls (you can adjust this)
+concurrent_calls = 150
+
+# List to store the futures of concurrent calls
+futures = []
+
+# Make concurrent calls
+with concurrent.futures.ThreadPoolExecutor(max_workers=concurrent_calls) as executor:
+ for _ in range(concurrent_calls):
+ random_question = random.choice(questions)
+ futures.append(executor.submit(make_openai_completion, random_question))
+
+# Wait for all futures to complete
+concurrent.futures.wait(futures)
+
+# Summarize the results
+successful_calls = 0
+failed_calls = 0
+
+for future in futures:
+ if future.result() is not None:
+ successful_calls += 1
+ else:
+ failed_calls += 1
+
+print("Load test Summary:")
+print(f"Total Requests: {concurrent_calls}")
+print(f"Successful Calls: {successful_calls}")
+print(f"Failed Calls: {failed_calls}")
+
+# Display content of the logs
+with open("request_log.txt", "r") as log_file:
+ print("\nRequest Log:\n", log_file.read())
+
+with open("error_log.txt", "r") as error_log_file:
+ print("\nError Log:\n", error_log_file.read())
diff --git a/cookbook/litellm_router/request_log.txt b/cookbook/litellm_router/request_log.txt
new file mode 100644
index 00000000..821d87ab
--- /dev/null
+++ b/cookbook/litellm_router/request_log.txt
@@ -0,0 +1,48 @@
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Response ID: 71a47cd4-92d9-4091-9429-8d22af6b56bf Url: /queue/response/71a47cd4-92d9-4091-9429-8d22af6b56bf
+Time: 0.77 seconds
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Response ID: a0855c20-59ba-4eed-85c1-e0719eebdeab Url: /queue/response/a0855c20-59ba-4eed-85c1-e0719eebdeab
+Time: 1.46 seconds
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Response ID: b131cdcd-0693-495b-ad41-b0cf2afc4833 Url: /queue/response/b131cdcd-0693-495b-ad41-b0cf2afc4833
+Time: 2.13 seconds
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Response ID: a58e5185-90e7-4832-9f28-e5a5ac167a40 Url: /queue/response/a58e5185-90e7-4832-9f28-e5a5ac167a40
+Time: 2.83 seconds
+
+Question: Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format.
+Response ID: 52dbbd49-eedb-4c11-8382-3ca7deb1af35 Url: /queue/response/52dbbd49-eedb-4c11-8382-3ca7deb1af35
+Time: 3.50 seconds
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Response ID: eedda05f-61e1-4081-b49d-27f9449bcf69 Url: /queue/response/eedda05f-61e1-4081-b49d-27f9449bcf69
+Time: 4.20 seconds
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Response ID: 8a484722-66ec-4193-b19b-2dfc4265cfd2 Url: /queue/response/8a484722-66ec-4193-b19b-2dfc4265cfd2
+Time: 4.89 seconds
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Response ID: ae1e2b71-d711-456d-8df0-13ce0709eb04 Url: /queue/response/ae1e2b71-d711-456d-8df0-13ce0709eb04
+Time: 5.60 seconds
+
+Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 10
+Response ID: cfabd174-838e-4252-b82b-648923573db8 Url: /queue/response/cfabd174-838e-4252-b82b-648923573db8
+Time: 6.29 seconds
+
+Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
+Response ID: 02d5b7d6-5443-41e9-94e4-90d8b00d49fb Url: /queue/response/02d5b7d6-5443-41e9-94e4-90d8b00d49fb
+Time: 7.01 seconds
+
diff --git a/cookbook/litellm_router/response_log.txt b/cookbook/litellm_router/response_log.txt
new file mode 100644
index 00000000..e69de29b
diff --git a/cookbook/litellm_router/test_questions/question1.txt b/cookbook/litellm_router/test_questions/question1.txt
new file mode 100644
index 00000000..d633a8ea
--- /dev/null
+++ b/cookbook/litellm_router/test_questions/question1.txt
@@ -0,0 +1,43 @@
+Given this context, what is litellm? LiteLLM about: About
+Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs). LiteLLM manages
+
+Translating inputs to the provider's completion and embedding endpoints
+Guarantees consistent output, text responses will always be available at ['choices'][0]['message']['content']
+Exception mapping - common exceptions across providers are mapped to the OpenAI exception types.
+10/05/2023: LiteLLM is adopting Semantic Versioning for all commits. Learn more
+10/16/2023: Self-hosted OpenAI-proxy server Learn more
+
+Usage (Docs)
+Important
+LiteLLM v1.0.0 is being launched to require openai>=1.0.0. Track this here
+
+Open In Colab
+pip install litellm
+from litellm import completion
+import os
+
+## set ENV variables
+os.environ["OPENAI_API_KEY"] = "your-openai-key"
+os.environ["COHERE_API_KEY"] = "your-cohere-key"
+
+messages = [{ "content": "Hello, how are you?","role": "user"}]
+
+# openai call
+response = completion(model="gpt-3.5-turbo", messages=messages)
+
+# cohere call
+response = completion(model="command-nightly", messages=messages)
+print(response)
+Streaming (Docs)
+liteLLM supports streaming the model response back, pass stream=True to get a streaming iterator in response.
+Streaming is supported for all models (Bedrock, Huggingface, TogetherAI, Azure, OpenAI, etc.)
+
+from litellm import completion
+response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
+for chunk in response:
+ print(chunk['choices'][0]['delta'])
+
+# claude 2
+result = completion('claude-2', messages, stream=True)
+for chunk in result:
+ print(chunk['choices'][0]['delta'])
\ No newline at end of file
diff --git a/cookbook/litellm_router/test_questions/question2.txt b/cookbook/litellm_router/test_questions/question2.txt
new file mode 100644
index 00000000..78188d06
--- /dev/null
+++ b/cookbook/litellm_router/test_questions/question2.txt
@@ -0,0 +1,65 @@
+Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs). LiteLLM manages
+
+Translating inputs to the provider's completion and embedding endpoints
+Guarantees consistent output, text responses will always be available at ['choices'][0]['message']['content']
+Exception mapping - common exceptions across providers are mapped to the OpenAI exception types.
+10/05/2023: LiteLLM is adopting Semantic Versioning for all commits. Learn more
+10/16/2023: Self-hosted OpenAI-proxy server Learn more
+
+Usage (Docs)
+Important
+LiteLLM v1.0.0 is being launched to require openai>=1.0.0. Track this here
+
+Open In Colab
+pip install litellm
+from litellm import completion
+import os
+
+## set ENV variables
+os.environ["OPENAI_API_KEY"] = "your-openai-key"
+os.environ["COHERE_API_KEY"] = "your-cohere-key"
+
+messages = [{ "content": "Hello, how are you?","role": "user"}]
+
+# openai call
+response = completion(model="gpt-3.5-turbo", messages=messages)
+
+# cohere call
+response = completion(model="command-nightly", messages=messages)
+print(response)
+Streaming (Docs)
+liteLLM supports streaming the model response back, pass stream=True to get a streaming iterator in response.
+Streaming is supported for all models (Bedrock, Huggingface, TogetherAI, Azure, OpenAI, etc.)
+
+from litellm import completion
+response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
+for chunk in response:
+ print(chunk['choices'][0]['delta'])
+
+# claude 2
+result = completion('claude-2', messages, stream=True)
+for chunk in result:
+ print(chunk['choices'][0]['delta']) Supported LiteLLM providers Supported Provider (Docs)
+Provider Completion Streaming Async Completion Async Streaming
+openai ā
ā
ā
ā
+azure ā
ā
ā
ā
+aws - sagemaker ā
ā
ā
ā
+aws - bedrock ā
ā
ā
ā
+cohere ā
ā
ā
ā
+anthropic ā
ā
ā
ā
+huggingface ā
ā
ā
ā
+replicate ā
ā
ā
ā
+together_ai ā
ā
ā
ā
+openrouter ā
ā
ā
ā
+google - vertex_ai ā
ā
ā
ā
+google - palm ā
ā
ā
ā
+ai21 ā
ā
ā
ā
+baseten ā
ā
ā
ā
+vllm ā
ā
ā
ā
+nlp_cloud ā
ā
ā
ā
+aleph alpha ā
ā
ā
ā
+petals ā
ā
ā
ā
+ollama ā
ā
ā
ā
+deepinfra ā
ā
ā
ā
+perplexity-ai ā
ā
ā
ā
+anyscale ā
ā
ā
ā
\ No newline at end of file
diff --git a/cookbook/litellm_router/test_questions/question3.txt b/cookbook/litellm_router/test_questions/question3.txt
new file mode 100644
index 00000000..d6006f9c
--- /dev/null
+++ b/cookbook/litellm_router/test_questions/question3.txt
@@ -0,0 +1,50 @@
+What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
+LiteLLM Server manages:
+
+Calling 100+ LLMs Huggingface/Bedrock/TogetherAI/etc. in the OpenAI ChatCompletions & Completions format
+Set custom prompt templates + model-specific configs (temperature, max_tokens, etc.)
+Quick Start
+View all the supported args for the Proxy CLI here
+
+$ litellm --model huggingface/bigcode/starcoder
+
+#INFO: Proxy running on http://0.0.0.0:8000
+
+Test
+In a new shell, run, this will make an openai.ChatCompletion request
+
+litellm --test
+
+This will now automatically route any requests for gpt-3.5-turbo to bigcode starcoder, hosted on huggingface inference endpoints.
+
+Replace openai base
+import openai
+
+openai.api_base = "http://0.0.0.0:8000"
+
+print(openai.chat.completions.create(model="test", messages=[{"role":"user", "content":"Hey!"}]))
+
+Supported LLMs
+Bedrock
+Huggingface (TGI)
+Anthropic
+VLLM
+OpenAI Compatible Server
+TogetherAI
+Replicate
+Petals
+Palm
+Azure OpenAI
+AI21
+Cohere
+$ export AWS_ACCESS_KEY_ID=""
+$ export AWS_REGION_NAME="" # e.g. us-west-2
+$ export AWS_SECRET_ACCESS_KEY=""
+
+$ litellm --model bedrock/anthropic.claude-v2
+
+Server Endpoints
+POST /chat/completions - chat completions endpoint to call 100+ LLMs
+POST /completions - completions endpoint
+POST /embeddings - embedding endpoint for Azure, OpenAI, Huggingface endpoints
+GET /models - available models on server
\ No newline at end of file
diff --git a/cookbook/litellm_router_load_test/memory_usage/router_endpoint.py b/cookbook/litellm_router_load_test/memory_usage/router_endpoint.py
new file mode 100644
index 00000000..1dc2d914
--- /dev/null
+++ b/cookbook/litellm_router_load_test/memory_usage/router_endpoint.py
@@ -0,0 +1,65 @@
+from fastapi import FastAPI
+import uvicorn
+from memory_profiler import profile
+import os
+import litellm
+from litellm import Router
+from dotenv import load_dotenv
+from litellm._uuid import uuid
+
+load_dotenv()
+
+model_list = [
+ {
+ "model_name": "gpt-3.5-turbo",
+ "litellm_params": {
+ "model": "azure/chatgpt-v-2",
+ "api_key": os.getenv("AZURE_API_KEY"),
+ "api_version": os.getenv("AZURE_API_VERSION"),
+ "api_base": os.getenv("AZURE_API_BASE"),
+ },
+ "tpm": 240000,
+ "rpm": 1800,
+ },
+ {
+ "model_name": "text-embedding-ada-002",
+ "litellm_params": {
+ "model": "azure/azure-embedding-model",
+ "api_key": os.getenv("AZURE_API_KEY"),
+ "api_base": os.getenv("AZURE_API_BASE"),
+ },
+ "tpm": 100000,
+ "rpm": 10000,
+ },
+]
+
+litellm.set_verbose = True
+litellm.cache = litellm.Cache(
+ type="s3", s3_bucket_name="litellm-my-test-bucket-2", s3_region_name="us-east-1"
+)
+router = Router(model_list=model_list, set_verbose=True)
+
+app = FastAPI()
+
+
+@app.get("/")
+async def read_root():
+ return {"message": "Welcome to the FastAPI endpoint!"}
+
+
+@profile
+@app.post("/router_acompletion")
+async def router_acompletion():
+ question = f"This is a test: {uuid.uuid4()}" * 100
+ resp = await router.aembedding(model="text-embedding-ada-002", input=question)
+ print("embedding-resp", resp)
+
+ response = await router.acompletion(
+ model="gpt-3.5-turbo", messages=[{"role": "user", "content": question}]
+ )
+ print("completion-resp", response)
+ return response
+
+
+if __name__ == "__main__":
+ uvicorn.run(app, host="0.0.0.0", port=8000)
diff --git a/cookbook/litellm_router_load_test/memory_usage/router_memory_usage copy.py b/cookbook/litellm_router_load_test/memory_usage/router_memory_usage copy.py
new file mode 100644
index 00000000..76d5d391
--- /dev/null
+++ b/cookbook/litellm_router_load_test/memory_usage/router_memory_usage copy.py
@@ -0,0 +1,91 @@
+#### What this tests ####
+
+from memory_profiler import profile
+import sys
+import os
+import time
+import asyncio
+
+sys.path.insert(
+ 0, os.path.abspath("../..")
+) # Adds the parent directory to the system path
+import litellm
+from litellm import Router
+from dotenv import load_dotenv
+from litellm._uuid import uuid
+
+load_dotenv()
+
+
+model_list = [
+ {
+ "model_name": "gpt-3.5-turbo", # openai model name
+ "litellm_params": { # params for litellm completion/embedding call
+ "model": "azure/chatgpt-v-2",
+ "api_key": os.getenv("AZURE_API_KEY"),
+ "api_version": os.getenv("AZURE_API_VERSION"),
+ "api_base": os.getenv("AZURE_API_BASE"),
+ },
+ "tpm": 240000,
+ "rpm": 1800,
+ },
+ {
+ "model_name": "text-embedding-ada-002",
+ "litellm_params": {
+ "model": "azure/azure-embedding-model",
+ "api_key": os.environ["AZURE_API_KEY"],
+ "api_base": os.environ["AZURE_API_BASE"],
+ },
+ "tpm": 100000,
+ "rpm": 10000,
+ },
+]
+litellm.set_verbose = True
+litellm.cache = litellm.Cache(
+ type="s3", s3_bucket_name="litellm-my-test-bucket-2", s3_region_name="us-east-1"
+)
+router = Router(
+ model_list=model_list,
+ set_verbose=True,
+) # type: ignore
+
+
+@profile
+async def router_acompletion():
+ # embedding call
+ question = f"This is a test: {uuid.uuid4()}" * 100
+ resp = await router.aembedding(model="text-embedding-ada-002", input=question)
+ print("embedding-resp", resp)
+
+ response = await router.acompletion(
+ model="gpt-3.5-turbo", messages=[{"role": "user", "content": question}]
+ )
+ print("completion-resp", response)
+ return response
+
+
+async def main():
+ for i in range(1):
+ start = time.time()
+ n = 50 # Number of concurrent tasks
+ tasks = [router_acompletion() for _ in range(n)]
+
+ chat_completions = await asyncio.gather(*tasks)
+
+ successful_completions = [c for c in chat_completions if c is not None]
+
+ # Write errors to error_log.txt
+ with open("error_log.txt", "a") as error_log:
+ for completion in chat_completions:
+ if isinstance(completion, str):
+ error_log.write(completion + "\n")
+
+ print(n, time.time() - start, len(successful_completions))
+ time.sleep(10)
+
+
+if __name__ == "__main__":
+ # Blank out contents of error_log.txt
+ open("error_log.txt", "w").close()
+
+ asyncio.run(main())
diff --git a/cookbook/litellm_router_load_test/memory_usage/router_memory_usage.py b/cookbook/litellm_router_load_test/memory_usage/router_memory_usage.py
new file mode 100644
index 00000000..76d5d391
--- /dev/null
+++ b/cookbook/litellm_router_load_test/memory_usage/router_memory_usage.py
@@ -0,0 +1,91 @@
+#### What this tests ####
+
+from memory_profiler import profile
+import sys
+import os
+import time
+import asyncio
+
+sys.path.insert(
+ 0, os.path.abspath("../..")
+) # Adds the parent directory to the system path
+import litellm
+from litellm import Router
+from dotenv import load_dotenv
+from litellm._uuid import uuid
+
+load_dotenv()
+
+
+model_list = [
+ {
+ "model_name": "gpt-3.5-turbo", # openai model name
+ "litellm_params": { # params for litellm completion/embedding call
+ "model": "azure/chatgpt-v-2",
+ "api_key": os.getenv("AZURE_API_KEY"),
+ "api_version": os.getenv("AZURE_API_VERSION"),
+ "api_base": os.getenv("AZURE_API_BASE"),
+ },
+ "tpm": 240000,
+ "rpm": 1800,
+ },
+ {
+ "model_name": "text-embedding-ada-002",
+ "litellm_params": {
+ "model": "azure/azure-embedding-model",
+ "api_key": os.environ["AZURE_API_KEY"],
+ "api_base": os.environ["AZURE_API_BASE"],
+ },
+ "tpm": 100000,
+ "rpm": 10000,
+ },
+]
+litellm.set_verbose = True
+litellm.cache = litellm.Cache(
+ type="s3", s3_bucket_name="litellm-my-test-bucket-2", s3_region_name="us-east-1"
+)
+router = Router(
+ model_list=model_list,
+ set_verbose=True,
+) # type: ignore
+
+
+@profile
+async def router_acompletion():
+ # embedding call
+ question = f"This is a test: {uuid.uuid4()}" * 100
+ resp = await router.aembedding(model="text-embedding-ada-002", input=question)
+ print("embedding-resp", resp)
+
+ response = await router.acompletion(
+ model="gpt-3.5-turbo", messages=[{"role": "user", "content": question}]
+ )
+ print("completion-resp", response)
+ return response
+
+
+async def main():
+ for i in range(1):
+ start = time.time()
+ n = 50 # Number of concurrent tasks
+ tasks = [router_acompletion() for _ in range(n)]
+
+ chat_completions = await asyncio.gather(*tasks)
+
+ successful_completions = [c for c in chat_completions if c is not None]
+
+ # Write errors to error_log.txt
+ with open("error_log.txt", "a") as error_log:
+ for completion in chat_completions:
+ if isinstance(completion, str):
+ error_log.write(completion + "\n")
+
+ print(n, time.time() - start, len(successful_completions))
+ time.sleep(10)
+
+
+if __name__ == "__main__":
+ # Blank out contents of error_log.txt
+ open("error_log.txt", "w").close()
+
+ asyncio.run(main())
diff --git a/cookbook/litellm_router_load_test/memory_usage/send_request.py b/cookbook/litellm_router_load_test/memory_usage/send_request.py
new file mode 100644
index 00000000..6a3473e2
--- /dev/null
+++ b/cookbook/litellm_router_load_test/memory_usage/send_request.py
@@ -0,0 +1,28 @@
+import requests
+from concurrent.futures import ThreadPoolExecutor
+
+# Replace the URL with your actual endpoint
+url = "http://localhost:8000/router_acompletion"
+
+
+def make_request(session):
+ headers = {"Content-Type": "application/json"}
+ data = {} # Replace with your JSON payload if needed
+
+ response = session.post(url, headers=headers, json=data)
+ print(f"Status code: {response.status_code}")
+
+
+# Number of concurrent requests
+num_requests = 20
+
+# Create a session to reuse the underlying TCP connection
+with requests.Session() as session:
+ # Use ThreadPoolExecutor for concurrent requests
+ with ThreadPoolExecutor(max_workers=num_requests) as executor:
+ # Use list comprehension to submit tasks
+ futures = [executor.submit(make_request, session) for _ in range(num_requests)]
+
+ # Wait for all futures to complete
+ for future in futures:
+ future.result()
diff --git a/cookbook/litellm_router_load_test/test_loadtest_openai_client.py b/cookbook/litellm_router_load_test/test_loadtest_openai_client.py
new file mode 100644
index 00000000..8c50825b
--- /dev/null
+++ b/cookbook/litellm_router_load_test/test_loadtest_openai_client.py
@@ -0,0 +1,73 @@
+import sys
+import os
+from dotenv import load_dotenv
+
+load_dotenv()
+sys.path.insert(
+ 0, os.path.abspath("../..")
+) # Adds the parent directory to the system path
+import asyncio
+from litellm import Timeout
+import time
+import openai
+
+### Test just calling AsyncAzureOpenAI
+
+openai_client = openai.AsyncAzureOpenAI(
+ azure_endpoint=os.getenv("AZURE_API_BASE"),
+ api_key=os.getenv("AZURE_API_KEY"),
+)
+
+
+async def call_acompletion(semaphore, input_data):
+ async with semaphore:
+ try:
+ # Use asyncio.wait_for to set a timeout for the task
+ response = await openai_client.chat.completions.create(**input_data)
+ # Handle the response as needed
+ print(response)
+ return response
+ except Timeout:
+ print(f"Task timed out: {input_data}")
+ return None # You may choose to return something else or raise an exception
+
+
+async def main():
+ # Initialize the Router
+
+ # Create a semaphore with a capacity of 100
+ semaphore = asyncio.Semaphore(100)
+
+ # List to hold all task references
+ tasks = []
+ start_time_all_tasks = time.time()
+ # Launch 1000 tasks
+ for _ in range(500):
+ task = asyncio.create_task(
+ call_acompletion(
+ semaphore,
+ {
+ "model": "chatgpt-v-2",
+ "messages": [{"role": "user", "content": "Hey, how's it going?"}],
+ },
+ )
+ )
+ tasks.append(task)
+
+ # Wait for all tasks to complete
+ responses = await asyncio.gather(*tasks)
+ # Process responses as needed
+ # Record the end time for all tasks
+ end_time_all_tasks = time.time()
+ # Calculate the total time for all tasks
+ total_time_all_tasks = end_time_all_tasks - start_time_all_tasks
+ print(f"Total time for all tasks: {total_time_all_tasks} seconds")
+
+ # Calculate the average time per response
+ average_time_per_response = total_time_all_tasks / len(responses)
+ print(f"Average time per response: {average_time_per_response} seconds")
+ print(f"NUMBER OF COMPLETED TASKS: {len(responses)}")
+
+
+# Run the main function
+asyncio.run(main())
diff --git a/cookbook/litellm_router_load_test/test_loadtest_router.py b/cookbook/litellm_router_load_test/test_loadtest_router.py
new file mode 100644
index 00000000..280e495e
--- /dev/null
+++ b/cookbook/litellm_router_load_test/test_loadtest_router.py
@@ -0,0 +1,87 @@
+import sys
+import os
+from dotenv import load_dotenv
+
+load_dotenv()
+sys.path.insert(
+ 0, os.path.abspath("../..")
+) # Adds the parent directory to the system path
+import asyncio
+from litellm import Router, Timeout
+import time
+
+### Test calling router async
+
+
+async def call_acompletion(semaphore, router: Router, input_data):
+ async with semaphore:
+ try:
+ # Use asyncio.wait_for to set a timeout for the task
+ response = await router.acompletion(**input_data)
+ # Handle the response as needed
+ print(response)
+ return response
+ except Timeout:
+ print(f"Task timed out: {input_data}")
+ return None # You may choose to return something else or raise an exception
+
+
+async def main():
+ # Initialize the Router
+ model_list = [
+ {
+ "model_name": "gpt-3.5-turbo",
+ "litellm_params": {
+ "model": "gpt-3.5-turbo",
+ "api_key": os.getenv("OPENAI_API_KEY"),
+ },
+ },
+ {
+ "model_name": "gpt-3.5-turbo",
+ "litellm_params": {
+ "model": "azure/chatgpt-v-2",
+ "api_key": os.getenv("AZURE_API_KEY"),
+ "api_base": os.getenv("AZURE_API_BASE"),
+ "api_version": os.getenv("AZURE_API_VERSION"),
+ },
+ },
+ ]
+ router = Router(model_list=model_list, num_retries=3, timeout=10)
+
+ # Create a semaphore with a capacity of 100
+ semaphore = asyncio.Semaphore(100)
+
+ # List to hold all task references
+ tasks = []
+ start_time_all_tasks = time.time()
+ # Launch 1000 tasks
+ for _ in range(500):
+ task = asyncio.create_task(
+ call_acompletion(
+ semaphore,
+ router,
+ {
+ "model": "gpt-3.5-turbo",
+ "messages": [{"role": "user", "content": "Hey, how's it going?"}],
+ },
+ )
+ )
+ tasks.append(task)
+
+ # Wait for all tasks to complete
+ responses = await asyncio.gather(*tasks)
+ # Process responses as needed
+ # Record the end time for all tasks
+ end_time_all_tasks = time.time()
+ # Calculate the total time for all tasks
+ total_time_all_tasks = end_time_all_tasks - start_time_all_tasks
+ print(f"Total time for all tasks: {total_time_all_tasks} seconds")
+
+ # Calculate the average time per response
+ average_time_per_response = total_time_all_tasks / len(responses)
+ print(f"Average time per response: {average_time_per_response} seconds")
+ print(f"NUMBER OF COMPLETED TASKS: {len(responses)}")
+
+
+# Run the main function
+asyncio.run(main())
diff --git a/cookbook/litellm_router_load_test/test_loadtest_router_withs3_cache.py b/cookbook/litellm_router_load_test/test_loadtest_router_withs3_cache.py
new file mode 100644
index 00000000..b093489b
--- /dev/null
+++ b/cookbook/litellm_router_load_test/test_loadtest_router_withs3_cache.py
@@ -0,0 +1,93 @@
+import sys
+import os
+from dotenv import load_dotenv
+
+load_dotenv()
+sys.path.insert(
+ 0, os.path.abspath("../..")
+) # Adds the parent directory to the system path
+import asyncio
+from litellm import Router, Timeout
+import time
+from litellm.caching.caching import Cache
+import litellm
+
+litellm.cache = Cache(
+ type="s3", s3_bucket_name="cache-bucket-litellm", s3_region_name="us-west-2"
+)
+
+### Test calling router with s3 Cache
+
+
+async def call_acompletion(semaphore, router: Router, input_data):
+ async with semaphore:
+ try:
+ # Use asyncio.wait_for to set a timeout for the task
+ response = await router.acompletion(**input_data)
+ # Handle the response as needed
+ print(response)
+ return response
+ except Timeout:
+ print(f"Task timed out: {input_data}")
+ return None # You may choose to return something else or raise an exception
+
+
+async def main():
+ # Initialize the Router
+ model_list = [
+ {
+ "model_name": "gpt-3.5-turbo",
+ "litellm_params": {
+ "model": "gpt-3.5-turbo",
+ "api_key": os.getenv("OPENAI_API_KEY"),
+ },
+ },
+ {
+ "model_name": "gpt-3.5-turbo",
+ "litellm_params": {
+ "model": "azure/chatgpt-v-2",
+ "api_key": os.getenv("AZURE_API_KEY"),
+ "api_base": os.getenv("AZURE_API_BASE"),
+ "api_version": os.getenv("AZURE_API_VERSION"),
+ },
+ },
+ ]
+ router = Router(model_list=model_list, num_retries=3, timeout=10)
+
+ # Create a semaphore with a capacity of 100
+ semaphore = asyncio.Semaphore(100)
+
+ # List to hold all task references
+ tasks = []
+ start_time_all_tasks = time.time()
+ # Launch 1000 tasks
+ for _ in range(500):
+ task = asyncio.create_task(
+ call_acompletion(
+ semaphore,
+ router,
+ {
+ "model": "gpt-3.5-turbo",
+ "messages": [{"role": "user", "content": "Hey, how's it going?"}],
+ },
+ )
+ )
+ tasks.append(task)
+
+ # Wait for all tasks to complete
+ responses = await asyncio.gather(*tasks)
+ # Process responses as needed
+ # Record the end time for all tasks
+ end_time_all_tasks = time.time()
+ # Calculate the total time for all tasks
+ total_time_all_tasks = end_time_all_tasks - start_time_all_tasks
+ print(f"Total time for all tasks: {total_time_all_tasks} seconds")
+
+ # Calculate the average time per response
+ average_time_per_response = total_time_all_tasks / len(responses)
+ print(f"Average time per response: {average_time_per_response} seconds")
+ print(f"NUMBER OF COMPLETED TASKS: {len(responses)}")
+
+
+# Run the main function
+asyncio.run(main())
diff --git a/cookbook/litellm_test_multiple_llm_demo.ipynb b/cookbook/litellm_test_multiple_llm_demo.ipynb
new file mode 100644
index 00000000..f22448e4
--- /dev/null
+++ b/cookbook/litellm_test_multiple_llm_demo.ipynb
@@ -0,0 +1,55 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "provenance": []
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "adotBkqZSh5g"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install litellm"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from litellm import completion\n",
+ "\n",
+ "## set ENV variables\n",
+ "os.environ[\"OPENAI_API_KEY\"] = \"openai key\"\n",
+ "os.environ[\"COHERE_API_KEY\"] = \"cohere key\"\n",
+ "os.environ[\"REPLICATE_API_KEY\"] = \"replicate key\"\n",
+ "messages = [{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}]\n",
+ "\n",
+ "# openai call\n",
+ "response = completion(model=\"gpt-3.5-turbo\", messages=messages)\n",
+ "\n",
+ "# cohere call\n",
+ "response = completion(\"command-nightly\", messages)\n",
+ "\n",
+ "# replicate call\n",
+ "response = completion(\"replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1\", messages)"
+ ],
+ "metadata": {
+ "id": "LeOqznSgSj-z"
+ },
+ "execution_count": null,
+ "outputs": []
+ }
+ ]
+}
diff --git a/cookbook/livekit_agent_sdk/README.md b/cookbook/livekit_agent_sdk/README.md
new file mode 100644
index 00000000..1c3f0bf9
--- /dev/null
+++ b/cookbook/livekit_agent_sdk/README.md
@@ -0,0 +1,114 @@
+# LiveKit Voice Agent with LiteLLM Gateway
+
+Simple example showing how to use LiveKit's xAI realtime plugin with LiteLLM as a proxy. This lets you switch between xAI, OpenAI, and Azure realtime APIs without changing your code.
+
+## Quick Start
+
+### 1. Install dependencies
+
+```bash
+pip install livekit-agents[xai] websockets
+```
+
+### 2. Start LiteLLM proxy
+
+```bash
+# With xAI
+export XAI_API_KEY="your-xai-key"
+litellm --config config.yaml --port 4000
+```
+
+### 3. Run the voice agent
+
+```bash
+python main.py
+```
+
+Type your message and get a voice response from Grok!
+
+## Configuration
+
+Set these environment variables if needed:
+
+```bash
+export LITELLM_PROXY_URL="http://localhost:4000"
+export LITELLM_API_KEY="sk-1234"
+export LITELLM_MODEL="grok-voice-agent"
+```
+
+Or use the defaults - connects to `http://localhost:4000` by default.
+
+## Example Config File
+
+Create a `config.yaml` with your realtime models:
+
+```yaml
+model_list:
+ - model_name: grok-voice-agent
+ litellm_params:
+ model: xai/grok-2-vision-1212
+ api_key: os.environ/XAI_API_KEY
+ model_info:
+ mode: realtime
+
+ - model_name: openai-voice-agent
+ litellm_params:
+ model: gpt-4o-realtime-preview
+ api_key: os.environ/OPENAI_API_KEY
+ model_info:
+ mode: realtime
+
+general_settings:
+ master_key: sk-1234
+```
+
+Then start: `litellm --config config.yaml --port 4000`
+
+## How It Works
+
+LiveKit's xAI plugin connects through LiteLLM proxy by setting `base_url`:
+
+```python
+from livekit.plugins import xai
+
+model = xai.realtime.RealtimeModel(
+ voice="ara",
+ api_key="sk-1234", # LiteLLM proxy key
+ base_url="http://localhost:4000", # Point to LiteLLM
+)
+```
+
+## Switching Providers
+
+Just change the model in your config - no code changes needed:
+
+**xAI Grok:**
+```yaml
+model: xai/grok-2-vision-1212
+```
+
+**OpenAI:**
+```yaml
+model: gpt-4o-realtime-preview
+```
+
+**Azure OpenAI:**
+```yaml
+model: azure/gpt-4o-realtime-preview
+api_base: https://your-endpoint.openai.azure.com/
+```
+
+## Why Use LiteLLM?
+
+- ā
**Switch providers** without changing agent code
+- ā
**Cost tracking** across all voice sessions
+- ā
**Rate limiting** and budgets
+- ā
**Load balancing** across multiple API keys
+- ā
**Fallbacks** to backup models
+
+## Learn More
+
+- [LiveKit xAI Realtime Tutorial](/docs/tutorials/livekit_xai_realtime)
+- [xAI Realtime Docs](/docs/providers/xai_realtime)
+- [LiveKit Agents Documentation](https://docs.livekit.io/agents/)
+- [LiteLLM Realtime API](/docs/realtime)
diff --git a/cookbook/livekit_agent_sdk/config.example.yaml b/cookbook/livekit_agent_sdk/config.example.yaml
new file mode 100644
index 00000000..1361f36a
--- /dev/null
+++ b/cookbook/livekit_agent_sdk/config.example.yaml
@@ -0,0 +1,21 @@
+model_list:
+ - model_name: grok-voice-agent
+ litellm_params:
+ model: xai/grok-2-vision-1212
+ api_key: os.environ/XAI_API_KEY
+ model_info:
+ mode: realtime
+
+ - model_name: openai-voice-agent
+ litellm_params:
+ model: gpt-4o-realtime-preview
+ api_key: os.environ/OPENAI_API_KEY
+ model_info:
+ mode: realtime
+
+litellm_settings:
+ drop_params: True
+ telemetry: False
+
+general_settings:
+ master_key: sk-1234 # Change this to a secure key
diff --git a/cookbook/livekit_agent_sdk/main.py b/cookbook/livekit_agent_sdk/main.py
new file mode 100644
index 00000000..c68e5534
--- /dev/null
+++ b/cookbook/livekit_agent_sdk/main.py
@@ -0,0 +1,121 @@
+"""
+Simple xAI Voice Agent using LiveKit SDK with LiteLLM Gateway
+
+This example shows how to use LiveKit's xAI realtime plugin through LiteLLM proxy.
+LiteLLM acts as a unified interface, allowing you to switch between xAI, OpenAI,
+and Azure realtime APIs without changing your agent code.
+"""
+
+import asyncio
+import json
+import os
+import websockets
+
+# Configuration
+PROXY_URL = os.getenv("LITELLM_PROXY_URL", "http://localhost:4000")
+API_KEY = os.getenv("LITELLM_API_KEY", "sk-1234")
+MODEL = os.getenv("LITELLM_MODEL", "grok-voice-agent")
+
+
+async def run_voice_agent():
+ """
+ Simple voice agent that:
+ 1. Connects to xAI realtime API through LiteLLM proxy
+ 2. Sends a user message
+ 3. Streams back the response
+ """
+
+ url = f"ws://{PROXY_URL.replace('http://', '').replace('https://', '')}/v1/realtime?model={MODEL}"
+ headers = {"Authorization": f"Bearer {API_KEY}"}
+
+ print(f"šļø Connecting to voice agent...")
+ print(f" Model: {MODEL}")
+ print(f" Proxy: {PROXY_URL}")
+ print()
+
+ async with websockets.connect(url, additional_headers=headers) as ws:
+ # Receive initial connection event
+ initial = json.loads(await ws.recv())
+ print(f"ā
Connected! Event: {initial['type']}\n")
+
+ # Get user input
+ user_message = input("š¬ Your message: ").strip()
+ if not user_message:
+ user_message = "Tell me a fun fact about AI!"
+
+ print(f"\nš¤ Sending to {MODEL}...\n")
+
+ # Send user message
+ await ws.send(
+ json.dumps(
+ {
+ "type": "conversation.item.create",
+ "item": {
+ "type": "message",
+ "role": "user",
+ "content": [{"type": "input_text", "text": user_message}],
+ },
+ }
+ )
+ )
+
+ # Request response
+ await ws.send(
+ json.dumps(
+ {
+ "type": "response.create",
+ "response": {"modalities": ["text", "audio"]},
+ }
+ )
+ )
+
+ # Stream response
+ print("š¤ Response: ", end="", flush=True)
+ transcript = []
+
+ try:
+ while True:
+ msg = await asyncio.wait_for(ws.recv(), timeout=15.0)
+ event = json.loads(msg)
+
+ # Capture transcript deltas
+ if event["type"] == "response.output_audio_transcript.delta":
+ delta = event.get("delta", "")
+ if delta:
+ print(delta, end="", flush=True)
+ transcript.append(delta)
+
+ # Done when response completes
+ elif event["type"] == "response.done":
+ break
+
+ except asyncio.TimeoutError:
+ pass
+
+ print("\n")
+
+ if transcript:
+ print(f"ā
Complete response: {''.join(transcript)}")
+
+ await ws.close()
+
+
+def main():
+ """Run the voice agent"""
+ print("=" * 70)
+ print("LiveKit xAI Voice Agent via LiteLLM Proxy")
+ print("=" * 70)
+ print()
+
+ try:
+ asyncio.run(run_voice_agent())
+ except KeyboardInterrupt:
+ print("\n\nš Goodbye!")
+ except Exception as e:
+ print(f"\nā Error: {e}")
+ print("\nMake sure LiteLLM proxy is running:")
+ print(f" litellm --config config.yaml --port 4000")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/cookbook/livekit_agent_sdk/requirements.txt b/cookbook/livekit_agent_sdk/requirements.txt
new file mode 100644
index 00000000..9e3542fa
--- /dev/null
+++ b/cookbook/livekit_agent_sdk/requirements.txt
@@ -0,0 +1,2 @@
+livekit-agents[xai]>=1.3.12
+websockets>=15.0.1
diff --git a/cookbook/logging_observability/LiteLLM_Arize.ipynb b/cookbook/logging_observability/LiteLLM_Arize.ipynb
new file mode 100644
index 00000000..72a082f8
--- /dev/null
+++ b/cookbook/logging_observability/LiteLLM_Arize.ipynb
@@ -0,0 +1,172 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4FbDOmcj2VkM"
+ },
+ "source": [
+ "## Use LiteLLM with Arize\n",
+ "https://docs.litellm.ai/docs/observability/arize_integration\n",
+ "\n",
+ "This method uses the litellm proxy to send the data to Arize. The callback is set in the litellm config below, instead of using OpenInference tracing."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "21W8Woog26Ns"
+ },
+ "source": [
+ "## Install Dependencies"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "id": "xrjKLBxhxu2L"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Requirement already satisfied: litellm in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (1.54.1)\n",
+ "Requirement already satisfied: aiohttp in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (3.11.10)\n",
+ "Requirement already satisfied: click in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (8.1.7)\n",
+ "Requirement already satisfied: httpx<0.28.0,>=0.23.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (0.27.2)\n",
+ "Requirement already satisfied: importlib-metadata>=6.8.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (8.5.0)\n",
+ "Requirement already satisfied: jinja2<4.0.0,>=3.1.2 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (3.1.4)\n",
+ "Requirement already satisfied: jsonschema<5.0.0,>=4.22.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (4.23.0)\n",
+ "Requirement already satisfied: openai>=1.55.3 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (1.57.1)\n",
+ "Requirement already satisfied: pydantic<3.0.0,>=2.0.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (2.10.3)\n",
+ "Requirement already satisfied: python-dotenv>=0.2.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (1.0.1)\n",
+ "Requirement already satisfied: requests<3.0.0,>=2.31.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (2.32.3)\n",
+ "Requirement already satisfied: tiktoken>=0.7.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (0.7.0)\n",
+ "Requirement already satisfied: tokenizers in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (0.21.0)\n",
+ "Requirement already satisfied: anyio in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from httpx<0.28.0,>=0.23.0->litellm) (4.7.0)\n",
+ "Requirement already satisfied: certifi in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from httpx<0.28.0,>=0.23.0->litellm) (2024.8.30)\n",
+ "Requirement already satisfied: httpcore==1.* in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from httpx<0.28.0,>=0.23.0->litellm) (1.0.7)\n",
+ "Requirement already satisfied: idna in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from httpx<0.28.0,>=0.23.0->litellm) (3.10)\n",
+ "Requirement already satisfied: sniffio in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from httpx<0.28.0,>=0.23.0->litellm) (1.3.1)\n",
+ "Requirement already satisfied: h11<0.15,>=0.13 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from httpcore==1.*->httpx<0.28.0,>=0.23.0->litellm) (0.14.0)\n",
+ "Requirement already satisfied: zipp>=3.20 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from importlib-metadata>=6.8.0->litellm) (3.21.0)\n",
+ "Requirement already satisfied: MarkupSafe>=2.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from jinja2<4.0.0,>=3.1.2->litellm) (3.0.2)\n",
+ "Requirement already satisfied: attrs>=22.2.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.22.0->litellm) (24.2.0)\n",
+ "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.22.0->litellm) (2024.10.1)\n",
+ "Requirement already satisfied: referencing>=0.28.4 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.22.0->litellm) (0.35.1)\n",
+ "Requirement already satisfied: rpds-py>=0.7.1 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.22.0->litellm) (0.22.3)\n",
+ "Requirement already satisfied: distro<2,>=1.7.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from openai>=1.55.3->litellm) (1.9.0)\n",
+ "Requirement already satisfied: jiter<1,>=0.4.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from openai>=1.55.3->litellm) (0.6.1)\n",
+ "Requirement already satisfied: tqdm>4 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from openai>=1.55.3->litellm) (4.67.1)\n",
+ "Requirement already satisfied: typing-extensions<5,>=4.11 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from openai>=1.55.3->litellm) (4.12.2)\n",
+ "Requirement already satisfied: annotated-types>=0.6.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from pydantic<3.0.0,>=2.0.0->litellm) (0.7.0)\n",
+ "Requirement already satisfied: pydantic-core==2.27.1 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from pydantic<3.0.0,>=2.0.0->litellm) (2.27.1)\n",
+ "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from requests<3.0.0,>=2.31.0->litellm) (3.4.0)\n",
+ "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from requests<3.0.0,>=2.31.0->litellm) (2.0.7)\n",
+ "Requirement already satisfied: regex>=2022.1.18 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from tiktoken>=0.7.0->litellm) (2024.11.6)\n",
+ "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from aiohttp->litellm) (2.4.4)\n",
+ "Requirement already satisfied: aiosignal>=1.1.2 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from aiohttp->litellm) (1.3.1)\n",
+ "Requirement already satisfied: frozenlist>=1.1.1 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from aiohttp->litellm) (1.5.0)\n",
+ "Requirement already satisfied: multidict<7.0,>=4.5 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from aiohttp->litellm) (6.1.0)\n",
+ "Requirement already satisfied: propcache>=0.2.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from aiohttp->litellm) (0.2.1)\n",
+ "Requirement already satisfied: yarl<2.0,>=1.17.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from aiohttp->litellm) (1.18.3)\n",
+ "Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from tokenizers->litellm) (0.26.5)\n",
+ "Requirement already satisfied: filelock in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers->litellm) (3.16.1)\n",
+ "Requirement already satisfied: fsspec>=2023.5.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers->litellm) (2024.10.0)\n",
+ "Requirement already satisfied: packaging>=20.9 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers->litellm) (24.2)\n",
+ "Requirement already satisfied: pyyaml>=5.1 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers->litellm) (6.0.2)\n"
+ ]
+ }
+ ],
+ "source": [
+ "!pip install litellm"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jHEu-TjZ29PJ"
+ },
+ "source": [
+ "## Set Env Variables"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "id": "QWd9rTysxsWO"
+ },
+ "outputs": [],
+ "source": [
+ "import litellm\n",
+ "import os\n",
+ "from getpass import getpass\n",
+ "\n",
+ "os.environ[\"ARIZE_SPACE_KEY\"] = getpass(\"Enter your Arize space key: \")\n",
+ "os.environ[\"ARIZE_API_KEY\"] = getpass(\"Enter your Arize API key: \")\n",
+ "os.environ['OPENAI_API_KEY']= getpass(\"Enter your OpenAI API key: \")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's run a completion call and see the traces in Arize"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Hello! Nice to meet you, OpenAI. How can I assist you today?\n"
+ ]
+ }
+ ],
+ "source": [
+ "# set arize as a callback, litellm will send the data to arize\n",
+ "litellm.callbacks = [\"arize\"]\n",
+ " \n",
+ "# openai call\n",
+ "response = litellm.completion(\n",
+ " model=\"gpt-3.5-turbo\",\n",
+ " messages=[\n",
+ " {\"role\": \"user\", \"content\": \"Hi š - i'm openai\"}\n",
+ " ]\n",
+ ")\n",
+ "print(response.choices[0].message.content)"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": ".venv",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/cookbook/logging_observability/LiteLLM_Langfuse.ipynb b/cookbook/logging_observability/LiteLLM_Langfuse.ipynb
new file mode 100644
index 00000000..2a63666e
--- /dev/null
+++ b/cookbook/logging_observability/LiteLLM_Langfuse.ipynb
@@ -0,0 +1,197 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "provenance": []
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Use LiteLLM with Langfuse\n",
+ "https://docs.litellm.ai/docs/observability/langfuse_integration"
+ ],
+ "metadata": {
+ "id": "4FbDOmcj2VkM"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Install Dependencies"
+ ],
+ "metadata": {
+ "id": "21W8Woog26Ns"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!pip install litellm langfuse"
+ ],
+ "metadata": {
+ "id": "xrjKLBxhxu2L"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Set Env Variables"
+ ],
+ "metadata": {
+ "id": "jHEu-TjZ29PJ"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "id": "QWd9rTysxsWO"
+ },
+ "outputs": [],
+ "source": [
+ "import litellm\n",
+ "from litellm import completion\n",
+ "import os\n",
+ "\n",
+ "# from https://cloud.langfuse.com/\n",
+ "os.environ[\"LANGFUSE_PUBLIC_KEY\"] = \"\"\n",
+ "os.environ[\"LANGFUSE_SECRET_KEY\"] = \"\"\n",
+ "\n",
+ "\n",
+ "# OpenAI and Cohere keys\n",
+ "# You can use any of the litellm supported providers: https://docs.litellm.ai/docs/providers\n",
+ "os.environ['OPENAI_API_KEY']=\"\"\n",
+ "os.environ['COHERE_API_KEY']=\"\"\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Set LangFuse as a callback for sending data\n",
+ "## OpenAI completion call"
+ ],
+ "metadata": {
+ "id": "NodQl0hp3Lma"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# set langfuse as a callback, litellm will send the data to langfuse\n",
+ "litellm.success_callback = [\"langfuse\"]\n",
+ "\n",
+ "# openai call\n",
+ "response = completion(\n",
+ " model=\"gpt-3.5-turbo\",\n",
+ " messages=[\n",
+ " {\"role\": \"user\", \"content\": \"Hi š - i'm openai\"}\n",
+ " ]\n",
+ ")\n",
+ "\n",
+ "print(response)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "vNAuwJY1yp_F",
+ "outputId": "c3a71e26-13f5-4379-fac9-409290ba79bb"
+ },
+ "execution_count": 8,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "{\n",
+ " \"id\": \"chatcmpl-85nP4xHdAP3jAcGneIguWATS9qdoO\",\n",
+ " \"object\": \"chat.completion\",\n",
+ " \"created\": 1696392238,\n",
+ " \"model\": \"gpt-3.5-turbo-0613\",\n",
+ " \"choices\": [\n",
+ " {\n",
+ " \"index\": 0,\n",
+ " \"message\": {\n",
+ " \"role\": \"assistant\",\n",
+ " \"content\": \"Hello! How can I assist you today?\"\n",
+ " },\n",
+ " \"finish_reason\": \"stop\"\n",
+ " }\n",
+ " ],\n",
+ " \"usage\": {\n",
+ " \"prompt_tokens\": 15,\n",
+ " \"completion_tokens\": 9,\n",
+ " \"total_tokens\": 24\n",
+ " }\n",
+ "}\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# we set langfuse as a callback in the prev cell\n",
+ "# cohere call\n",
+ "response = completion(\n",
+ " model=\"command-nightly\",\n",
+ " messages=[\n",
+ " {\"role\": \"user\", \"content\": \"Hi š - i'm cohere\"}\n",
+ " ]\n",
+ ")\n",
+ "\n",
+ "print(response)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "2PMSLc_FziJL",
+ "outputId": "1c37605e-b406-4ffc-aafd-e1983489c6be"
+ },
+ "execution_count": 9,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "{\n",
+ " \"object\": \"chat.completion\",\n",
+ " \"choices\": [\n",
+ " {\n",
+ " \"finish_reason\": \"stop\",\n",
+ " \"index\": 0,\n",
+ " \"message\": {\n",
+ " \"content\": \" Nice to meet you, Cohere! I'm excited to be meeting new members of the AI community\",\n",
+ " \"role\": \"assistant\",\n",
+ " \"logprobs\": null\n",
+ " }\n",
+ " }\n",
+ " ],\n",
+ " \"id\": \"chatcmpl-a14e903f-4608-4ceb-b996-8ebdf21360ca\",\n",
+ " \"created\": 1696392247.3313863,\n",
+ " \"model\": \"command-nightly\",\n",
+ " \"usage\": {\n",
+ " \"prompt_tokens\": 8,\n",
+ " \"completion_tokens\": 20,\n",
+ " \"total_tokens\": 28\n",
+ " }\n",
+ "}\n"
+ ]
+ }
+ ]
+ }
+ ]
+}
\ No newline at end of file
diff --git a/cookbook/logging_observability/LiteLLM_Lunary.ipynb b/cookbook/logging_observability/LiteLLM_Lunary.ipynb
new file mode 100644
index 00000000..3b1dc5d5
--- /dev/null
+++ b/cookbook/logging_observability/LiteLLM_Lunary.ipynb
@@ -0,0 +1,348 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4FbDOmcj2VkM"
+ },
+ "source": [
+ "## Use LiteLLM with Langfuse\n",
+ "https://docs.litellm.ai/docs/observability/langfuse_integration"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "21W8Woog26Ns"
+ },
+ "source": [
+ "## Install Dependencies"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "xrjKLBxhxu2L"
+ },
+ "outputs": [],
+ "source": [
+ "%pip install litellm lunary"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jHEu-TjZ29PJ"
+ },
+ "source": [
+ "## Set Env Variables"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "id": "QWd9rTysxsWO"
+ },
+ "outputs": [],
+ "source": [
+ "import litellm\n",
+ "from litellm import completion\n",
+ "import os\n",
+ "\n",
+ "# from https://app.lunary.ai/\n",
+ "os.environ[\"LUNARY_PUBLIC_KEY\"] = \"\"\n",
+ "\n",
+ "\n",
+ "# LLM provider keys\n",
+ "# You can use any of the litellm supported providers: https://docs.litellm.ai/docs/providers\n",
+ "os.environ['OPENAI_API_KEY'] = \"\"\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "NodQl0hp3Lma"
+ },
+ "source": [
+ "## Set Lunary as a callback for sending data\n",
+ "## OpenAI completion call"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "vNAuwJY1yp_F",
+ "outputId": "c3a71e26-13f5-4379-fac9-409290ba79bb"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[Choices(finish_reason='stop', index=0, message=Message(content='Hello! How can I assist you today?', role='assistant'))]ModelResponse(id='chatcmpl-8xIWykI0GiJSmYtXYuB8Z363kpIBm', choices=[Choices(finish_reason='stop', index=0, message=Message(content='Hello! How can I assist you today?', role='assistant'))], created=1709143276, model='gpt-3.5-turbo-0125', object='chat.completion', system_fingerprint='fp_86156a94a0', usage=Usage(completion_tokens=9, prompt_tokens=15, total_tokens=24))\n",
+ "\n",
+ "[Lunary] Add event: {\n",
+ " \"event\": \"start\",\n",
+ " \"type\": \"llm\",\n",
+ " \"name\": \"gpt-3.5-turbo\",\n",
+ " \"runId\": \"a363776a-bd07-4474-bce2-193067f01b2e\",\n",
+ " \"timestamp\": \"2024-02-28T18:01:15.188153+00:00\",\n",
+ " \"input\": {\n",
+ " \"role\": \"user\",\n",
+ " \"content\": \"Hi \\ud83d\\udc4b - i'm openai\"\n",
+ " },\n",
+ " \"extra\": {},\n",
+ " \"runtime\": \"litellm\",\n",
+ " \"metadata\": {}\n",
+ "}\n",
+ "\n",
+ "\n",
+ "[Lunary] Add event: {\n",
+ " \"event\": \"end\",\n",
+ " \"type\": \"llm\",\n",
+ " \"runId\": \"a363776a-bd07-4474-bce2-193067f01b2e\",\n",
+ " \"timestamp\": \"2024-02-28T18:01:16.846581+00:00\",\n",
+ " \"output\": {\n",
+ " \"role\": \"assistant\",\n",
+ " \"content\": \"Hello! How can I assist you today?\"\n",
+ " },\n",
+ " \"runtime\": \"litellm\",\n",
+ " \"tokensUsage\": {\n",
+ " \"completion\": 9,\n",
+ " \"prompt\": 15\n",
+ " }\n",
+ "}\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "--- Logging error ---\n",
+ "Traceback (most recent call last):\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 537, in _make_request\n",
+ " response = conn.getresponse()\n",
+ " ^^^^^^^^^^^^^^^^^^\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/connection.py\", line 466, in getresponse\n",
+ " httplib_response = super().getresponse()\n",
+ " ^^^^^^^^^^^^^^^^^^^^^\n",
+ " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py\", line 1423, in getresponse\n",
+ " response.begin()\n",
+ " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py\", line 331, in begin\n",
+ " version, status, reason = self._read_status()\n",
+ " ^^^^^^^^^^^^^^^^^^^\n",
+ " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py\", line 292, in _read_status\n",
+ " line = str(self.fp.readline(_MAXLINE + 1), \"iso-8859-1\")\n",
+ " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
+ " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/socket.py\", line 707, in readinto\n",
+ " return self._sock.recv_into(b)\n",
+ " ^^^^^^^^^^^^^^^^^^^^^^^\n",
+ "TimeoutError: timed out\n",
+ "\n",
+ "The above exception was the direct cause of the following exception:\n",
+ "\n",
+ "Traceback (most recent call last):\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/requests/adapters.py\", line 486, in send\n",
+ " resp = conn.urlopen(\n",
+ " ^^^^^^^^^^^^^\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 847, in urlopen\n",
+ " retries = retries.increment(\n",
+ " ^^^^^^^^^^^^^^^^^^\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/util/retry.py\", line 470, in increment\n",
+ " raise reraise(type(error), error, _stacktrace)\n",
+ " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/util/util.py\", line 39, in reraise\n",
+ " raise value\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 793, in urlopen\n",
+ " response = self._make_request(\n",
+ " ^^^^^^^^^^^^^^^^^^^\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 539, in _make_request\n",
+ " self._raise_timeout(err=e, url=url, timeout_value=read_timeout)\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 370, in _raise_timeout\n",
+ " raise ReadTimeoutError(\n",
+ "urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=3333): Read timed out. (read timeout=5)\n",
+ "\n",
+ "During handling of the above exception, another exception occurred:\n",
+ "\n",
+ "Traceback (most recent call last):\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/lunary/consumer.py\", line 59, in send_batch\n",
+ " response = requests.post(\n",
+ " ^^^^^^^^^^^^^^\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/requests/api.py\", line 115, in post\n",
+ " return request(\"post\", url, data=data, json=json, **kwargs)\n",
+ " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/requests/api.py\", line 59, in request\n",
+ " return session.request(method=method, url=url, **kwargs)\n",
+ " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/requests/sessions.py\", line 589, in request\n",
+ " resp = self.send(prep, **send_kwargs)\n",
+ " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/requests/sessions.py\", line 703, in send\n",
+ " r = adapter.send(request, **kwargs)\n",
+ " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/requests/adapters.py\", line 532, in send\n",
+ " raise ReadTimeout(e, request=request)\n",
+ "requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=3333): Read timed out. (read timeout=5)\n",
+ "\n",
+ "During handling of the above exception, another exception occurred:\n",
+ "\n",
+ "Traceback (most recent call last):\n",
+ " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py\", line 1160, in emit\n",
+ " msg = self.format(record)\n",
+ " ^^^^^^^^^^^^^^^^^^^\n",
+ " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py\", line 999, in format\n",
+ " return fmt.format(record)\n",
+ " ^^^^^^^^^^^^^^^^^^\n",
+ " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py\", line 703, in format\n",
+ " record.message = record.getMessage()\n",
+ " ^^^^^^^^^^^^^^^^^^^\n",
+ " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py\", line 392, in getMessage\n",
+ " msg = msg % self.args\n",
+ " ~~~~^~~~~~~~~~~\n",
+ "TypeError: not all arguments converted during string formatting\n",
+ "Call stack:\n",
+ " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py\", line 1030, in _bootstrap\n",
+ " self._bootstrap_inner()\n",
+ " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py\", line 1073, in _bootstrap_inner\n",
+ " self.run()\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/lunary/consumer.py\", line 24, in run\n",
+ " self.send_batch()\n",
+ " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/lunary/consumer.py\", line 73, in send_batch\n",
+ " logging.error(\"[Lunary] Error sending events\", e)\n",
+ "Message: '[Lunary] Error sending events'\n",
+ "Arguments: (ReadTimeout(ReadTimeoutError(\"HTTPConnectionPool(host='localhost', port=3333): Read timed out. (read timeout=5)\")),)\n"
+ ]
+ }
+ ],
+ "source": [
+ "# set langfuse as a callback, litellm will send the data to langfuse\n",
+ "litellm.success_callback = [\"lunary\"]\n",
+ "\n",
+ "# openai call\n",
+ "response = completion(\n",
+ " model=\"gpt-3.5-turbo\",\n",
+ " messages=[\n",
+ " {\"role\": \"user\", \"content\": \"Hi š - i'm openai\"}\n",
+ " ]\n",
+ ")\n",
+ "\n",
+ "print(response)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Using LiteLLM with Lunary Templates\n",
+ "\n",
+ "You can use LiteLLM seamlessly with Lunary templates to manage your prompts and completions.\n",
+ "\n",
+ "Assuming you have created a template \"test-template\" with a variable \"question\", you can use it like this:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "2PMSLc_FziJL",
+ "outputId": "1c37605e-b406-4ffc-aafd-e1983489c6be"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[Choices(finish_reason='stop', index=0, message=Message(content='Hello! How can I assist you today?', role='assistant'))]ModelResponse(id='chatcmpl-8xIXegwpudg4YKnLB6pmpFGXqTHcH', choices=[Choices(finish_reason='stop', index=0, message=Message(content='Hello! How can I assist you today?', role='assistant'))], created=1709143318, model='gpt-4-0125-preview', object='chat.completion', system_fingerprint='fp_c8aa5a06d6', usage=Usage(completion_tokens=9, prompt_tokens=21, total_tokens=30))\n",
+ "\n",
+ "[Lunary] Add event: {\n",
+ " \"event\": \"start\",\n",
+ " \"type\": \"llm\",\n",
+ " \"name\": \"gpt-4-turbo-preview\",\n",
+ " \"runId\": \"3a5b698d-cb55-4b3b-ab6d-04d2b99e40cb\",\n",
+ " \"timestamp\": \"2024-02-28T18:01:56.746249+00:00\",\n",
+ " \"input\": [\n",
+ " {\n",
+ " \"role\": \"system\",\n",
+ " \"content\": \"You are an helpful assistant.\"\n",
+ " },\n",
+ " {\n",
+ " \"role\": \"user\",\n",
+ " \"content\": \"Hi! Hello!\"\n",
+ " }\n",
+ " ],\n",
+ " \"extra\": {\n",
+ " \"temperature\": 1,\n",
+ " \"max_tokens\": 100\n",
+ " },\n",
+ " \"runtime\": \"litellm\",\n",
+ " \"metadata\": {}\n",
+ "}\n",
+ "\n",
+ "\n",
+ "[Lunary] Add event: {\n",
+ " \"event\": \"end\",\n",
+ " \"type\": \"llm\",\n",
+ " \"runId\": \"3a5b698d-cb55-4b3b-ab6d-04d2b99e40cb\",\n",
+ " \"timestamp\": \"2024-02-28T18:01:58.741244+00:00\",\n",
+ " \"output\": {\n",
+ " \"role\": \"assistant\",\n",
+ " \"content\": \"Hello! How can I assist you today?\"\n",
+ " },\n",
+ " \"runtime\": \"litellm\",\n",
+ " \"tokensUsage\": {\n",
+ " \"completion\": 9,\n",
+ " \"prompt\": 21\n",
+ " }\n",
+ "}\n",
+ "\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "import lunary\n",
+ "from litellm import completion\n",
+ "\n",
+ "template = lunary.render_template(\"test-template\", {\"question\": \"Hello!\"})\n",
+ "\n",
+ "response = completion(**template)\n",
+ "\n",
+ "print(response)"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.12.2"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/cookbook/logging_observability/LiteLLM_Proxy_Langfuse.ipynb b/cookbook/logging_observability/LiteLLM_Proxy_Langfuse.ipynb
new file mode 100644
index 00000000..0baaab3f
--- /dev/null
+++ b/cookbook/logging_observability/LiteLLM_Proxy_Langfuse.ipynb
@@ -0,0 +1,252 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## LLM Ops Stack - LiteLLM Proxy + Langfuse \n",
+ "\n",
+ "This notebook demonstrates how to use LiteLLM Proxy with Langfuse \n",
+ "- Use LiteLLM Proxy for calling 100+ LLMs in OpenAI format\n",
+ "- Use Langfuse for viewing request / response traces \n",
+ "\n",
+ "\n",
+ "In this notebook we will setup LiteLLM Proxy to make requests to OpenAI, Anthropic, Bedrock and automatically log traces to Langfuse."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 1. Setup LiteLLM Proxy\n",
+ "\n",
+ "### 1.1 Define .env variables \n",
+ "Define .env variables on the container that litellm proxy is running on.\n",
+ "```bash\n",
+ "## LLM API Keys\n",
+ "OPENAI_API_KEY=sk-proj-1234567890\n",
+ "ANTHROPIC_API_KEY=sk-ant-api03-1234567890\n",
+ "AWS_ACCESS_KEY_ID=1234567890\n",
+ "AWS_SECRET_ACCESS_KEY=1234567890\n",
+ "\n",
+ "## Langfuse Logging \n",
+ "LANGFUSE_PUBLIC_KEY=\"pk-lf-xxxx9\"\n",
+ "LANGFUSE_SECRET_KEY=\"sk-lf-xxxx9\"\n",
+ "LANGFUSE_HOST=\"https://us.cloud.langfuse.com\"\n",
+ "```\n",
+ "\n",
+ "\n",
+ "### 1.1 Setup LiteLLM Proxy Config yaml \n",
+ "```yaml\n",
+ "model_list:\n",
+ " - model_name: gpt-4o\n",
+ " litellm_params:\n",
+ " model: openai/gpt-4o\n",
+ " api_key: os.environ/OPENAI_API_KEY\n",
+ " - model_name: claude-3-5-sonnet-20241022\n",
+ " litellm_params:\n",
+ " model: anthropic/claude-3-5-sonnet-20241022\n",
+ " api_key: os.environ/ANTHROPIC_API_KEY\n",
+ " - model_name: us.amazon.nova-micro-v1:0\n",
+ " litellm_params:\n",
+ " model: bedrock/us.amazon.nova-micro-v1:0\n",
+ " aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID\n",
+ " aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY\n",
+ "\n",
+ "litellm_settings:\n",
+ " callbacks: [\"langfuse\"]\n",
+ "\n",
+ "\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 2. Make LLM Requests to LiteLLM Proxy\n",
+ "\n",
+ "Now we will make our first LLM request to LiteLLM Proxy"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 2.1 Setup Client Side Variables to point to LiteLLM Proxy\n",
+ "Set `LITELLM_PROXY_BASE_URL` to the base url of the LiteLLM Proxy and `LITELLM_VIRTUAL_KEY` to the virtual key you want to use for Authentication to LiteLLM Proxy. (Note: In this initial setup you can)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\n",
+ "LITELLM_PROXY_BASE_URL=\"http://0.0.0.0:4000\"\n",
+ "LITELLM_VIRTUAL_KEY=\"sk-oXXRa1xxxxxxxxxxx\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "ChatCompletion(id='chatcmpl-B0sq6QkOKNMJ0dwP3x7OoMqk1jZcI', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Langfuse is a platform designed to monitor, observe, and troubleshoot AI and large language model (LLM) applications. It provides features that help developers gain insights into how their AI systems are performing, make debugging easier, and optimize the deployment of models. Langfuse allows for tracking of model interactions, collecting telemetry, and visualizing data, which is crucial for understanding the behavior of AI models in production environments. This kind of tool is particularly useful for developers working with language models who need to ensure reliability and efficiency in their applications.', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1739550502, model='gpt-4o-2024-08-06', object='chat.completion', service_tier='default', system_fingerprint='fp_523b9b6e5f', usage=CompletionUsage(completion_tokens=109, prompt_tokens=13, total_tokens=122, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)))"
+ ]
+ },
+ "execution_count": 22,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import openai\n",
+ "client = openai.OpenAI(\n",
+ " api_key=LITELLM_VIRTUAL_KEY,\n",
+ " base_url=LITELLM_PROXY_BASE_URL\n",
+ ")\n",
+ "\n",
+ "response = client.chat.completions.create(\n",
+ " model=\"gpt-4o\",\n",
+ " messages = [\n",
+ " {\n",
+ " \"role\": \"user\",\n",
+ " \"content\": \"what is Langfuse?\"\n",
+ " }\n",
+ " ],\n",
+ ")\n",
+ "\n",
+ "response"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 2.3 View Traces on Langfuse\n",
+ "LiteLLM will send the request / response, model, tokens (input + output), cost to Langfuse.\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 2.4 Call Anthropic, Bedrock models \n",
+ "\n",
+ "Now we can call `us.amazon.nova-micro-v1:0` and `claude-3-5-sonnet-20241022` models defined on your config.yaml both in the OpenAI request / response format."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "ChatCompletion(id='chatcmpl-7756e509-e61f-4f5e-b5ae-b7a41013522a', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"Langfuse is an observability tool designed specifically for machine learning models and applications built with natural language processing (NLP) and large language models (LLMs). It focuses on providing detailed insights into how these models perform in real-world scenarios. Here are some key features and purposes of Langfuse:\\n\\n1. **Real-time Monitoring**: Langfuse allows developers to monitor the performance of their NLP and LLM applications in real time. This includes tracking the inputs and outputs of the models, as well as any errors or issues that arise during operation.\\n\\n2. **Error Tracking**: It helps in identifying and tracking errors in the models' outputs. By analyzing incorrect or unexpected responses, developers can pinpoint where and why errors occur, facilitating more effective debugging and improvement.\\n\\n3. **Performance Metrics**: Langfuse provides various performance metrics, such as latency, throughput, and error rates. These metrics help developers understand how well their models are performing under different conditions and workloads.\\n\\n4. **Traceability**: It offers detailed traceability of requests and responses, allowing developers to follow the path of a request through the system and see how it is processed by the model at each step.\\n\\n5. **User Feedback Integration**: Langfuse can integrate user feedback to provide context for model outputs. This helps in understanding how real users are interacting with the model and how its outputs align with user expectations.\\n\\n6. **Customizable Dashboards**: Users can create custom dashboards to visualize the data collected by Langfuse. These dashboards can be tailored to highlight the most important metrics and insights for a specific application or team.\\n\\n7. **Alerting and Notifications**: It can set up alerts for specific conditions or errors, notifying developers when something goes wrong or when performance metrics fall outside of acceptable ranges.\\n\\nBy providing comprehensive observability for NLP and LLM applications, Langfuse helps developers to build more reliable, accurate, and user-friendly models and services.\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1739554005, model='us.amazon.nova-micro-v1:0', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=380, prompt_tokens=5, total_tokens=385, completion_tokens_details=None, prompt_tokens_details=None))"
+ ]
+ },
+ "execution_count": 24,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import openai\n",
+ "client = openai.OpenAI(\n",
+ " api_key=LITELLM_VIRTUAL_KEY,\n",
+ " base_url=LITELLM_PROXY_BASE_URL\n",
+ ")\n",
+ "\n",
+ "response = client.chat.completions.create(\n",
+ " model=\"us.amazon.nova-micro-v1:0\",\n",
+ " messages = [\n",
+ " {\n",
+ " \"role\": \"user\",\n",
+ " \"content\": \"what is Langfuse?\"\n",
+ " }\n",
+ " ],\n",
+ ")\n",
+ "\n",
+ "response"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 3. Advanced - Set Langfuse Trace ID, Tags, Metadata \n",
+ "\n",
+ "Here is an example of how you can set Langfuse specific params on your client side request. See full list of supported langfuse params [here](https://docs.litellm.ai/docs/observability/langfuse_integration)\n",
+ "\n",
+ "You can view the logged trace of this request [here](https://us.cloud.langfuse.com/project/clvlhdfat0007vwb74m9lvfvi/traces/567890?timestamp=2025-02-14T17%3A30%3A26.709Z)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "ChatCompletion(id='chatcmpl-789babd5-c064-4939-9093-46e4cd2e208a', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"Langfuse is an observability platform designed specifically for monitoring and improving the performance of natural language processing (NLP) models and applications. It provides developers with tools to track, analyze, and optimize how their language models interact with users and handle natural language inputs.\\n\\nHere are some key features and benefits of Langfuse:\\n\\n1. **Real-Time Monitoring**: Langfuse allows developers to monitor their NLP applications in real time. This includes tracking user interactions, model responses, and overall performance metrics.\\n\\n2. **Error Tracking**: It helps in identifying and tracking errors in the model's responses. This can include incorrect, irrelevant, or unsafe outputs.\\n\\n3. **User Feedback Integration**: Langfuse enables the collection of user feedback directly within the platform. This feedback can be used to identify areas for improvement in the model's performance.\\n\\n4. **Performance Metrics**: The platform provides detailed metrics and analytics on model performance, including latency, throughput, and accuracy.\\n\\n5. **Alerts and Notifications**: Developers can set up alerts to notify them of any significant issues or anomalies in model performance.\\n\\n6. **Debugging Tools**: Langfuse offers tools to help developers debug and refine their models by providing insights into how the model processes different types of inputs.\\n\\n7. **Integration with Development Workflows**: It integrates seamlessly with various development environments and CI/CD pipelines, making it easier to incorporate observability into the development process.\\n\\n8. **Customizable Dashboards**: Users can create custom dashboards to visualize the data in a way that best suits their needs.\\n\\nLangfuse aims to help developers build more reliable, accurate, and user-friendly NLP applications by providing them with the tools to observe and improve how their models perform in real-world scenarios.\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1739554281, model='us.amazon.nova-micro-v1:0', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=346, prompt_tokens=5, total_tokens=351, completion_tokens_details=None, prompt_tokens_details=None))"
+ ]
+ },
+ "execution_count": 27,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import openai\n",
+ "client = openai.OpenAI(\n",
+ " api_key=LITELLM_VIRTUAL_KEY,\n",
+ " base_url=LITELLM_PROXY_BASE_URL\n",
+ ")\n",
+ "\n",
+ "response = client.chat.completions.create(\n",
+ " model=\"us.amazon.nova-micro-v1:0\",\n",
+ " messages = [\n",
+ " {\n",
+ " \"role\": \"user\",\n",
+ " \"content\": \"what is Langfuse?\"\n",
+ " }\n",
+ " ],\n",
+ " extra_body={\n",
+ " \"metadata\": {\n",
+ " \"generation_id\": \"1234567890\",\n",
+ " \"trace_id\": \"567890\",\n",
+ " \"trace_user_id\": \"user_1234567890\",\n",
+ " \"tags\": [\"tag1\", \"tag2\"]\n",
+ " }\n",
+ " }\n",
+ ")\n",
+ "\n",
+ "response"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## "
+ ]
+ }
+ ],
+ "metadata": {
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/cookbook/logging_observability/litellm_proxy_langfuse.png b/cookbook/logging_observability/litellm_proxy_langfuse.png
new file mode 100644
index 00000000..6b0691e6
Binary files /dev/null and b/cookbook/logging_observability/litellm_proxy_langfuse.png differ
diff --git a/cookbook/misc/RELEASE_NOTES_GENERATION_INSTRUCTIONS.md b/cookbook/misc/RELEASE_NOTES_GENERATION_INSTRUCTIONS.md
new file mode 100644
index 00000000..4a6fa936
--- /dev/null
+++ b/cookbook/misc/RELEASE_NOTES_GENERATION_INSTRUCTIONS.md
@@ -0,0 +1,543 @@
+# LiteLLM Release Notes Generation Instructions
+
+This document provides comprehensive instructions for AI agents to generate release notes for LiteLLM following the established format and style.
+
+## Required Inputs
+
+1. **Release Version** (e.g., `v1.77.3-stable`)
+2. **PR Diff/Changelog** - List of PRs with titles and contributors
+3. **Previous Version Commit Hash** - To compare model pricing changes
+4. **Reference Release Notes** - Use recent stable releases (v1.76.3-stable, v1.77.2-stable) as templates for consistent formatting
+
+### Resolving Staging PRs
+
+The GitHub release page (e.g. `https://github.com/BerriAI/litellm/releases/tag/v1.83.3-stable`) does **not** list the real changelog directly. The "What's Changed" section contains **staging PRs** that each bundle many individual commits/PRs. For example:
+
+- `Litellm oss staging 03 14 2026 by @RheagalFire in #23686`
+- `Litellm ryan march 16 by @ryan-crabbe in #23822`
+
+To get the real changelog, you MUST click into each staging PR (e.g. `#23686`, `#23822`), open its **Commits** tab, and extract every underlying commit/PR (look for the `(#NNNNN)` suffix on commit titles). Those underlying PRs ā not the staging PRs ā are what get categorized in the release notes. Never treat a staging PR title as a single changelog entry.
+
+**IMPORTANT ā staging PRs are not the complete source.** Some PRs land on the release branch *before* the staging PRs and are therefore not reachable via `gh api /pulls/