| title | API Reference |
|---|
Your API root will be the URL from your Freeplay instance plus the suffix /api/v2/
For example https://app.freeplay.ai/api/v2
If you are a private deployment, it may be something like https://acme.freeplay.ai/api/v2.
Freeplay authenticates your API request using your API Key which can be managed through the Freeplay application at https://app.freeplay.ai/settings/api-access
To authenticate, provide an API key in the Authorization header
```bash bash Authorization: Bearer {freeplay_api_key} ```When using the Freeplay HTTP API, you may encounter various error responses. We use standard HTTP error codes, such as 200 for success, 400 for bad requests, 401 for authorization errors, and 500 for server errors. Most errors will also include an HTTP body, encoded in JSON, describing what went wrong and how to fix the issue.
It is not helpful to retry errors with codes in the 400-499 range, because they are typically client errors and will not be fixed by future requests.
On the other hand, retrying requests that resulted in 500 error codes will sometimes work, for example in the case that a request was sent to an instance that was terminated or during a transient network issue. We recommend retrying requests that received a 500 response up to three times, after at least a 5 second delay.
Here's a list of common errors, their status codes, and explanations:
400 Bad request
This means that the request was malformed or had bad data. This could happen if the request is not proper JSON or if it contains invalid data.
```bash bash {"message": "Session ID 456 is not a valid uuid4 format."} ```401 Unauthorized
This response will be returned if an API request does not contain the correct bearer authentication with a valid API key.
```bash bash {"message": "Requests must be authenticated with a valid API key."} ```404 Not found
This response indicates one of the IDs passed in the request is invalid, and the object does not exist. This can happen if the referenced object was deleted, or if the caller does not have access to the object.
```bash bash {"message": "Project not found"} ```500 Server error
This response indicates an error in the Freeplay server. Our team is notified for issues of this type via PagerDuty. Sometimes these are transient issues that can be fixed by retrying the request after a delay.
```bash bash {"message": "An unexpected error occurred. Please try again later."} ```When encountering any of these errors, check your request parameters and payload to ensure they match the API specifications. If issues persist, contact Freeplay support for further assistance.
A project in Freeplay represents a meaningful separation between your AI products. Most commonly we see people split projects out by AI product for easier tracking. To get all of your projects in freeplay, you can hit the /projects/all endpoint.
Note: this does not work with project-scoped API keys!
Note: This will not return private projects
```bash bash curl --location 'https://app.freeplay.ai/api/v2/projects/allx' \ --header 'Authorization: Bearer ' \ --data ''{ "projects": [ { "id": "", "name": "FreeplayChat" }, { "id": "", "name": "Freeplay Support Agent" }, { "id": "", "name": "Getting Started Project" }, ... }
</CodeGroup>
# Prompt Templates
A [prompt template](/docs/managing-prompts) in the Freeplay platform is a pre-defined structure for creating prompts that are used to interact with LLMs.
**Base URL**: `/api/v2/projects/<project-id>/prompt-templates`
## Retrieve a Prompt Template by Name
`POST /name/<template-name>`
You can retrieve a Prompt Template in one of three forms: Raw, Bound, or Formatted. Look at sample response payloads below for field details.
**Raw** will give you the Prompt Template without Input Variable values inserted
**Bound** will give you the Prompt Template with Input Variable values inserted and with content in a consistent format. Pass your Input Variables in the request body to get a Bound Prompt.
**Formatted** (most common) will give you the Prompt Template with Input Variable values inserted *and* with content formatted according to your LLM provider. Pass format = true as a query parameter and pass your Input Variables in the request body to get a Formatted Prompt
**Query Parameters**
| Parameter | Type | Description | Required |
| ----------- | --------- | ---------------------------------------------------------------- | -------------------- |
| environment | `str` | The environment tag of the prompt you want to fetch | No `Default: latest` |
| format | `boolean` | Indicates whether to return a Formatted Prompt or a Bound Prompt | No |
**Retrieve a Formatted Prompt**
<CodeGroup>
```bash bash
curl 'https://app.freeplay.ai/api/v2/projects/8f93dd00-2eb5-4ba2-9354-86d5c6831dfd/prompt-templates/name/album_bot?environment=prod&format=true' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer TOKEN' \
--data '{
"pop_star": "justin beiber"
}'
# SAMPLE RESPONSE PAYLOAD
{
"format_version": 2,
"formatted_content": [
{
"content": "You are a helpful bot that generates album names\nGenerate a two word album name in the style of justin beiber",
"role": "user"
}
],
"formatted_tool_schema": [
{
"function": {
"description": "Get weather of a location",
"name": "weather_of_location",
"parameters": {
"properties": {
"location": {
"description": "Location to get the weather for",
"type": "string"
}
},
"required": [
"location"
],
"type": "object"
}
},
"type": "function"
}
],
"metadata": {
"flavor": "openai_chat",
"model": "gpt-3.5-turbo-0125",
"params": {
"max_tokens": 100,
"temperature": 0.2
},
"provider": "openai",
"provider_info": {}
},
"prompt_template_id": "5cccc8e6-b163-4094-8bd2-90030f151ec8",
"prompt_template_name": "album_bot",
"prompt_template_version_id": "f503c15e-2f0f-4ce4-b443-4c87d0b6435d",
"system_content": null
}
Retrieve a Bound Prompt
```bash bash curl 'https://app.freeplay.ai/api/v2/projects/8f93dd00-2eb5-4ba2-9354-86d5c6831dfd/prompt-templates/name/album_bot?environment=prod' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer TOKEN' \ --data '{ "pop_star": "justin beiber" }'{ "bound_content": [ { "content": "You are a helpful bot that generates album names\nGenerate a two word album name in the style of justin beiber", "role": "user" } ], "format_version": 2, "metadata": { "flavor": "openai_chat", "model": "gpt-3.5-turbo-0125", "params": { "max_tokens": 100, "temperature": 0.2 }, "provider": "openai", "provider_info": {} }, "prompt_template_id": "5cccc8e6-b163-4094-8bd2-90030f151ec8", "prompt_template_name": "album_bot", "prompt_template_version_id": "f503c15e-2f0f-4ce4-b443-4c87d0b6435d", "tool_schema": [ { "description": "Get weather of a location", "name": "weather_of_location", "parameters": { "additionalProperties": false, "properties": { "location": { "description": "Location to get the weather for", "type": "string" } }, "required": [ "location" ], "type": "object" } } ] }
</CodeGroup>
**Retrieve a Raw Prompt Template**
<CodeGroup>
```bash bash
curl 'https://app.freeplay.ai/api/v2/projects/8f93dd00-2eb5-4ba2-9354-86d5c6831dfd/prompt-templates/name/album_bot?environment=prod' \
--header 'Authorization: Bearer TOKEN' \
# SAMPLE RESPONSE PAYLOAD
{
"content": [
{
"content": "You are a helpful bot that generates album names\nGenerate a two word album name in the style of {{pop_star}}",
"role": "user"
}
],
"format_version": 2,
"metadata": {
"flavor": "openai_chat",
"model": "gpt-3.5-turbo-0125",
"params": {
"max_tokens": 100,
"temperature": 0.2
},
"provider": "openai",
"provider_info": {}
},
"prompt_template_id": "5cccc8e6-b163-4094-8bd2-90030f151ec8",
"prompt_template_name": "album_bot",
"prompt_template_version_id": "f503c15e-2f0f-4ce4-b443-4c87d0b6435d",
"tool_schema": [
{
"description": "Get weather of a location",
"name": "weather_of_location",
"parameters": {
"additionalProperties": false,
"properties": {
"location": {
"description": "Location to get the weather for",
"type": "string"
}
},
"required": [
"location"
],
"type": "object"
}
}
]
}
GET /all/<environment-name>
{ "prompt_templates": [ { "content": [ { "content": "You are a helpful bot that generates album names\nRespond with only the album name", "role": "system" }, { "content": "Generate a two word album name in the style of {{pop_star}}", "role": "user" } ], "format_version": 2, "metadata": { "flavor": "anthropic_chat", "model": "claude-3-opus-20240229", "params": { "max_tokens": 256, "temperature": 0.5 }, "provider": "anthropic", "provider_info": {} }, "prompt_template_id": "5cccc8e6-b163-4094-8bd2-90030f151ec8", "prompt_template_name": "album_bot", "prompt_template_version_id": "eb934c9f-ee1e-42be-a9cb-d193f5d3aa54" }, { "content": [ { "content": "You are a news article summarization agent. Your job is to find the big idea in an article, distill it into compelling content & write a catchy summary that people want to click. You will be provided with the article title, source url and the article text\nThe scraped source content of the URL for the article is provided. ", "role": "system" }, { "content": "Here is the scraped content from the web page:\n--------\n{{article_title}}\n{{article_text}}\n{{source_url}}\n--------\n\nSome additional instructions follow. Please follow these carefully.\n\nThere may be user comments at the bottom or other superfluous information included from the scraped content. Ignore those and focus on the body of the article. \n\nAt the end of each response, after a newline always include the original URL for the article so people can read the article. \n\nWhen sharing the URL, include it after two line breaks and use this format starting with "Read more" and stripping the http:// at the start of the url: Read more: www.example.com\nGiven this article, respond with a Tweet of no more than 200 characters. Never include hashtags. A relevant emoji is ok. Do not include quotes around any content.\n\nAfter a newline, include the original URL for the article so people can read the article.", "role": "user" } ], "format_version": 2, "metadata": { "flavor": "openai_chat", "model": "gpt-3.5-turbo-16k-0613", "params": { "max_tokens": 2000, "temperature": 0.5 }, "provider": "openai", "provider_info": {} }, "prompt_template_id": "9daa7233-16d3-48b3-a739-ddba66742b08", "prompt_template_name": "article_summarizer", "prompt_template_version_id": "9f9f9bfe-ed53-449d-a13c-5f88198c0f8e" } }
</CodeGroup>
## Retrieve a Prompt Template by Version Id
Retrieve a Prompt Template using a specific Version Id. This can be useful if you want to pin your code to a specific prompt version regardless of actions taken in the Freeplay app
`POST /id/<template-id>/versions/<template-version-id>`
The same Raw, Bound, and Formatted options exist when retrieving by version id as they do when [retrieving by name](/docs/api-reference#retrieve-a-prompt-template-by-name)
<CodeGroup>
```bash bash
curl 'https://app.freeplay.ai//api/v2/projects/8f93dd00-2eb5-4ba2-9354-86d5c6831dfd/prompt-templates/id/0053c449-c736-4557-a3d4-bf3b852d0604/versions/42d301b2-cdc3-48d5-829b-b892561017ff' \
--header 'Authorization: Bearer TOKEN'
# Sample response payload
{
"content": [
{
"content": "You are a technical customer support agent for a product called freeplay. Your job is to help users answer questions about freeplay.\nYou will often be provided with information that will help you answer the question. If the information is relevant then use it to answer the question and end your response with \"See <source> for more details\" where <source> is a valid HTML formatted hyperlink to the source URL.\n\nIf the information provided is not relevant then ignore it.\n\nUser Query: {{question}}\n----\nSupporting Information: {{supporting_information}}",
"role": "user"
}
],
"format_version": 2,
"metadata": {
"flavor": "openai_chat",
"model": "gpt-3.5-turbo-16k-0613",
"params": {
"max_tokens": 2000,
"temperature": 0.5
},
"provider": "openai",
"provider_info": {}
},
"prompt_template_id": "0053c449-c736-4557-a3d4-bf3b852d0604",
"prompt_template_name": "rag-qa",
"prompt_template_version_id": "42d301b2-cdc3-48d5-829b-b892561017ff"
}
POST /id/<template-id>/versions
Add a new version to an existing Prompt Template. The new version is immediately deployed to the latest environment unless you pass another environment.
| Parameter | Type | Description | Required |
|---|---|---|---|
content |
str |
JSON-encoded array of chat-style messages, each with role and content. Variables use {{variable}} syntax. |
✓ |
model |
str |
Model name (e.g. gpt-4o-2024-05-13). |
✓ |
provider |
str |
Provider key (openai, anthropic, vertex, …). |
✓ |
llm_parameters |
dict[str, number, bool] |
Overrides for model parameters (e.g. temperature, max_tokens). |
✓ |
tool_schema |
list[NormalizedToolSchema] |
JSON schema(s) for function / tool calls. Each name must be unique. | — |
version_name |
str |
Human-readable name shown in the UI. | — |
version_description |
str |
Longer description shown in the UI. | — |
environments |
list[str] |
Deploy this version to the following environments (defaults to ["latest"]). |
— |
Successful calls return 201 Created with the new version’s metadata.
```bash bash curl -X POST \ 'https://app.freeplay.ai/api/v2/projects/9cf924e8-4474-442a-bf9f-53f8415a4bd2/prompt-templates/id/d500b46e-0ae2-46c0-ac0e-0c18feb180fb/versions' \ --header 'Authorization: Bearer $TOKEN' \ --header 'Content-Type: application/json' \ --data '{ "content": "[{\"role\": \"system\", \"content\": \"You are a helpful assistant.\"}, {\"role\": \"user\", \"content\": \"Hello {{name}}!\"}]", "provider": "openai", "model": "gpt-4o-2024-05-13", "llm_parameters": { "temperature": 0.2, "max_tokens": 256 }, "tool_schema": [{ "name": "get_weather", "description": "Get weather information", "parameters": { "type": "object", "properties": { "location": { "type": "string" } }, "required": ["location"] } }], "version_name": "v2 – temp 0.2", "version_description": "Updated version with lower temperature", "tags": ["latest"] }' ``` ```bash bash { "content": [ { "content": "You are a helpful assistant.", "role": "system" }, { "content": "Hello {{name}}!", "role": "user" } ], "created_at": 1754585113, "format_version": 3, "metadata": { "flavor": "openai_chat", "model": "gpt-4o-2024-05-13", "params": { "max_tokens": 256, "temperature": 0.2 }, "provider": "openai", "provider_info": {} }, "project_id": "9cf924e8-4474-442a-bf9f-53f8415a4bd2", "prompt_template_id": "d500b46e-0ae2-46c0-ac0e-0c18feb180fb", "prompt_template_name": "testing234", "prompt_template_version_id": "3a583bb3-901e-4f78-8364-f23b2f260019", "tool_schema": [ { "description": "Get weather information", "name": "get_weather", "parameters": { "properties": { "location": { "type": "string" } }, "required": [ "location" ], "type": "object" } } ], "version_description": "Updated version with lower temperature", "version_name": "v2 – temp 0.2" } ```-
400 Bad Request
"Invalid prompt template content format …"–contentis not valid JSON or message list."No matching model found …"– unknownmodel/provider."This provider does not support tool calls …"– model lacks tool-use support."Duplicate tool name found …"– duplicate names intool_schema.
-
404 Not found –
template_iddoes not exist or you lack access.
POST /id/<template-id>/versions/<version-id>/environments
Assign an existing Prompt Template Version to one or more environments.
- Environments must already exist in your account.
- The
latestenvironment is managed by the system and is ignored if provided. - This operation is idempotent and additive: it ensures the version is tagged with the provided environments but does not remove any previously assigned environments.
| Parameter | Type | Description | Required |
|---|---|---|---|
environments |
list[string] |
Environment names to assign (e.g., ["staging", "prod"]). latest ignored. |
✓ |
{}
</CodeGroup>
##### Validation & Errors
* 400 Bad Request
* `"Environment '<name>' not found …"` – the environment does not exist for your account.
* 404 Not Found
* The `template_id` or `version_id` does not exist, or you lack access.
# Agents
An [Agent](/docs/agents) in Freeplay is a process in your application that makes use of multiple LLM calls to generate a single response.
**Base URL:** `/api/v2/projects/<project-id>/agents`
## List Agents
`GET /`
Agents will be returned in descending chronological order (newest first). You can use pagination to navigate through agents beyond the initial page.
**Parameters**
| Parameter | Type | Description | Required |
| ---------- | ------ | ---------------------------------------------------------------- | ---------------- |
| page | int | The page number of agents to retrieve. | No `Default: 1` |
| page\_size | int | The number of agents to include in the response. Maximum is 100. | No `Default: 30` |
| name | string | Filter agents by exact name match (case-sensitive). | No |
<CodeGroup>
```bash bash
curl --location 'https://app.freeplay.ai/api/v2/projects/8f93dd00-2eb5-4ba2-9354-86d5c6831dfd/agents' \
--header 'Authorization: Bearer TOKEN'
Sample Response
```bash bash { "data": [ { "id": "deadbeef-dead-0000-beef-deadbeefdead", "name": "customer_support_agent" }, { "id": "f00dbabe-f00d-1111-babe-f00dbabef00d", "name": "sales_assistant" }, ], "pagination": { "page": 1, "page_size": 30, "has_next": false } } ```Example with Filters
```bash bash # Get agents with a specific name curl --location 'https://app.freeplay.ai/api/v2/projects/8f93dd00-2eb5-4ba2-9354-86d5c6831dfd/agents?name=customer_support_agent' \ --header 'Authorization: Bearer TOKEN'curl --location 'https://app.freeplay.ai/api/v2/projects/8f93dd00-2eb5-4ba2-9354-86d5c6831dfd/agents?page=2&page_size=50'
--header 'Authorization: Bearer TOKEN'
</CodeGroup>
# Sessions
Sessions are a collection of 1 or more LLM completions — enabling you to tie completions together in whatever logical grouping makes sense for your application (multi-turn chat, chains, agent flows, etc.).
**Base URL:** `/api/v2/projects/<project-id>/sessions`
## Record a Completion
`POST /<session-id>/completions`
**Request Payload**
| Parameter | Type | Description | Required |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| messages | `list[{"role": Literal["user", "assistant"], "content": str}]` | List of messages send to the LLM.Note for Anthropic: messages must exclude message fields other than `role` and `content`. | Yes |
| tool\_schema | `list[{
name: str,
description: str,
parameters: dict[str, any]
}]` | Tool schema used with the completion | No |
| inputs | `dict[str, any]` | Input variables used to build the formatted prompt. Values must not be null. | Yes |
| session\_info | `custom_metatdata: dict[str, str]` | Information to be associated with the Session | No |
| prompt\_info | `prompt_template_version_id: UUID environment: str` | Information associated with the Prompt Template | Yes |
| call\_info | `start_time: float` `end_time: float ``model: float` `provider: float` `provider_info: dict[str, any]` `llm_parameters: dict[str, any]` `usage: {prompt_tokens: int, completion_tokens: int}` `api_style: 'batch' \| 'default'` | Information associated with the LLM call. Note if `usage` is included, the token count will be used directly to calculate costs. If it is empty, Freeplay will estimate token counts using Tiktoken | No |
| test\_run\_info | `test_run_id: UUID test_case_id: UUID` | Information for the associated Test Run | No |
| completion\_id | `UUID` | The UUIDv4 that should be used as the ID for the completion. Valid for sending completion feedback. | No |
| eval\_results | `dict[str, bool\|float]` | A set of code evals for a completion. These will be displayed alongside Model-graded evals and human labels for a completion. | No |
<CodeGroup>
```bash bash
curl 'https://app.freeplay.ai/api/v2/projects/8f93dd00-2eb5-4ba2-9354-86d5c6831dfd/sessions/f503c15e-2f0f-4ce4-b443-4c87d0b6435d/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer TOKEN' \
--data '{
"messages": [
{
"content": "You are a helpful bot that generates album names\nGenerate a two word album name in the style of Taylor Swift",
"role": "user"
},
{
"role": "assistant",
"content": "\"Rainy Melodies\""
}
],
"inputs": {
"pop_star": "Taylor Swift"
},
"prompt_info": {
"prompt_template_version_id": "f503c15e-2f0f-4ce4-b443-4c87d0b6435d",
"environment": "prod"
},
"eval_results": {
"includes_required_keyword": true,
"toxicity_score": 0.28
}
}
'
# SAMPLE RESPONSE PAYLOAD
{
"completion_id": "707bc301-85e9-4f02-aa97-faba8cd7774a"
}
Record an LLM Interaction End to End
```bash bash import requests from dotenv import load_dotenv import os import json import uuidload_dotenv("../.env") project_id = os.getenv("FREEPLAY_PROJECT_ID") freeplay_api_root = f"https://app.freeplay.ai/api/v2/projects/{project_id}" openai_api_url = "https://api.openai.com/v1/chat/completions"
prompt_name = "album_bot" input_variables = {"pop_star": "Taylor Swift"} prompt_req = requests.post( url=f"{freeplay_api_root}/prompt-templates/name/{prompt_name}", headers={ "Authorization" : f"Bearer {os.getenv('FREEPLAY_KEY')}" }, params={ "environment": "prod", "format": "true" }, data=json.dumps(input_variables) ) formatted_prompt = prompt_req.json()
completion_payload = { "model": formatted_prompt["metadata"]["model"], "messages": formatted_prompt["formatted_content"], **formatted_prompt["metadata"]["params"] }
completion_req = requests.post( url=openai_api_url, headers={ "Authorization" : f"Bearer {os.getenv('OPENAI_API_KEY')}", "Content-Type": "application/json" }, data=json.dumps(completion_payload) ) response_message = completion_req.json()['choices'][0]['message']
messages = formatted_prompt["formatted_content"] messages.append(response_message)
record_payload = { "messages": messages, "inputs": input_variables, "prompt_info": { "prompt_template_version_id": formatted_prompt["prompt_template_version_id"], "environment": "prod" } }
session_id = str(uuid.uuid4())
requests.post( url=f"{freeplay_api_root}/sessions/{session_id}/completions", headers={ "Authorization": f"Bearer {os.getenv('FREEPLAY_KEY')}", "Content-Type": "application/json" }, data = json.dumps(record_payload) )
</CodeGroup>
## Retrieve Existing Sessions/Completions
`GET /`
Retrieve sessions, including their metadata and completions, starting from most recent first.
**Parameters**
| Parameter | Type | Description | Required |
| ------------------- | ---- | -------------------------------------------------------------------------------------------------------------- | ---------------- |
| page | int | The page number of the sessions to retrieve. | No `Default: 1` |
| page\_size | int | The number of sessions to include in the request. **Maximum page size is 100.** | No `Default: 10` |
| from\_date | str | A date that is the earliest for which sessions will be returned (inclusive). Must be in the format YYYY-MM-DD. | No |
| to\_date | str | A date that is the latest for which sessions will be returned (exclusive). Must be in the format YYYY-MM-DD. | No |
| test\_list | str | Filters to sessions that are in the given dataset. Pass in the name of the dataset. | No |
| test\_run\_id | str | Filters to sessions that are in the given test run. | No |
| prompt\_name | str | Filters to sessions that are associated with the given prompt name. | No |
| review\_queue\_id | str | Filters to sessions that in the given review queue. | No |
| custom\_metadata.\* | dict | Filters to sessions that have the associated meta data value at the session level | No |
<CodeGroup>
```bash bash
curl --header "Authorization: Bearer $FREEPLAY_API_KEY" "https://app.freeplay.ai/api/v2/projects/$FREEPLAY_PROJECT_ID/sessions"
# Sample Response Payload
[
{
"custom_metadata": {},
"messages": [
{
"client_evaluation_results": [],
"completion_id": "0202ae57-1098-4d3e-94f7-1fbb034fbed5",
"customer_feedback": {},
"environment": "latest",
"evaluation_results": [],
"format_version": 3,
"input_variables": {
"question": "How do I bundle prompts so my system is reliable?"
},
"message_type": "completion",
"model_name": "claude-2.1",
"prompt": [
{
"content": "You are a technical customer support agent for a product called freeplay.",
"role": "system"
},
{
"content": "How do I bundle prompts so my system is reliable?",
"role": "user"
}
],
"prompt_template_name": "my-prompt",
"provider_name": "anthropic",
"response": "You bundle prompts by downloading them with the python SDK, packaging them with your application, and using the FilesystemTemplateResolver when using the Freeplay SDK.",
"trace_id": null
}
],
"session_id": "85d7d393-4e85-4664-8598-5dd91dc75b5b",
"start_time": "2024-07-05T14:33:02.721000"
}
]
The following example is showing off using the following filters:
from_date: 2025-08-01to_date: 2025-08-22page: 1page_size: 70 (default is 10, max is 100)custom_metadata.is_llm_only: false
DELETE /api/v2/projects/<project-id>/sessions/<session-id>
Test Runs in Freeplay provide a structured way to generate batch tests for your language models using saved Datasets managed on the Freeplay server.
Base URL: /api/v2/projects/<project-id>/test-runs
POST /
Request Payload
| Parameters | Type | Description | Required |
|---|---|---|---|
| dataset_name | str |
The name of the dataset for which you want to create a Test Run | Yes |
| include_outputs | boolean |
Determines if Outputs are returned for test cases | No Default: True |
| test_run_name | str |
Set a name for the Test Run, to be displayed in the Freeplay app | No |
| test_run_description | str |
Set a description for the Test Run, to be displayed in the Freeplay app | No |
{
"test_cases": [
{
"output": null,
"test_case_id": "91e60c9e-fbaa-4990-b4cc-7a8bd067f298",
"variables": {
"question": "How do test runs relate to test cases?",
"supporting_information": "([SearchChunk(title='Test Runs', link='https://docs.freeplay.ai/docs/test-runs-with-freeplay', content='Test Runs')..."
}
},
{
"output": null,
"test_case_id": "c45dbc01-a177-449b-ab66-bb5a469748fe",
"variables": {
"question": "Write me some typescript code to run a new Test Run in freeplay",
"supporting_information": "[{"id":88,"version":5,"score":0.8668302,"payload":{"description":"Ship better products with LLMs...."
}
}
],
"test_run_description": "",
"test_run_id": "bd3eb06c-f93b-46a4-aa3b-d240789c8a06",
"test_run_name": ""
}
</CodeGroup>
**Execute a Test Run end to end**
<CodeGroup>
```bash bash
import requests
from dotenv import load_dotenv
import os
import json
import uuid
load_dotenv("../.env")
project_id = "45e20940-322a-4cb8-810f-3934a56aa5e2"
freeplay_api_root = f"https://app.freeplay.ai/api/v2/projects/{project_id}"
openai_api_url = "https://api.openai.com/v1/chat/completions"
# create the test run
test_run_req = requests.post(
url= freeplay_api_root + "/test-runs",
headers={
"Authorization": f"Bearer {os.getenv('FREEPLAY_KEY')}",
"Content-Type": "application/json"
},
data=json.dumps({
"dataset_name": "Example Tests"
})
)
test_run = test_run_req.json()
prompt_name = "rag-qa"
# loop over each test case in the dataset
for test_case in test_run["test_cases"]:
# fetch the formatted prompt
prompt_req = requests.post(
url=f"{freeplay_api_root}/prompt-templates/name/{prompt_name}",
headers={
"Authorization" : f"Bearer {os.getenv('FREEPLAY_KEY')}"
},
params={
"environment": "prod",
"format": "true"
},
data=json.dumps(test_case['variables'])
)
formatted_prompt = prompt_req.json()
# execute the completion
completion_payload = {
"model": formatted_prompt["metadata"]["model"],
"messages": formatted_prompt["formatted_content"],
**formatted_prompt["metadata"]["params"]
}
completion_req = requests.post(
url=openai_api_url,
headers={
"Authorization" : f"Bearer {os.getenv('OPENAI_API_KEY')}",
"Content-Type": "application/json"
},
data=json.dumps(completion_payload)
)
response_message = completion_req.json()['choices'][0]['message']
# record to freeplay
messages = formatted_prompt["formatted_content"]
messages.append(response_message)
record_payload = {
"messages": messages,
"inputs": test_case['variables'],
"prompt_info": {
"prompt_template_version_id": formatted_prompt["prompt_template_version_id"],
"environment": "prod"
},
# include the test run info to link the session to the test run
"test_run_info": {
"test_run_id": test_run["test_run_id"],
"test_case_id": test_case["test_case_id"]
}
}
# create a session
session_id = str(uuid.uuid4())
record_res = requests.post(
url=f"{freeplay_api_root}/sessions/{session_id}/completions",
headers={
"Authorization": f"Bearer {os.getenv('FREEPLAY_KEY')}",
"Content-Type": "application/json"
},
data=json.dumps(record_payload)
)
GET /id/<test-run-id>
{ "created_at": 1733770879, "description": "", "id": "2a9dd8bd-6c29-47c8-9ca4-427f73174881", "model_name": "gpt-4o-mini-2024-07-18", "name": "132--after-concurrency", "prompt_name": "rag-qa", "prompt_version": "gpt-4o-mini", "sessions_count": 132, "summary_statistics": { "auto_evaluation": { "Answer Accuracy": { "1": 104, "2": 9, "4": 9, "5": 10 }, "Answer Faithfulness": { "no": 126, "yes": 6 }, "Context Relevance": { "1": 116, "4": 2, "5": 14 } }, "client_evaluation": {}, "human_evaluation": {} } }
</CodeGroup>
## List Test Runs
`GET /`
Test Run will be returned in descending chronologic order. You can use pagination to get to test runs not in the most recent 100.
**Parameters**
| Parameter | Type | Description | Required |
| ---------- | ---- | ---------------------------------------------------------------------- | ----------------- |
| page | int | The page number of the sessions to retrieve. Maximum page size is 100. | No `Default: 1` |
| page\_size | int | The number of sessions to include in the request. | No `Default: 100` |
<CodeGroup>
```bash bash
curl --location 'https://app.freeplay.ai/api/v2/projects/45e20940-322a-4cb8-810f-3934a56aa5e2/test-runs' \
--header 'Authorization: Bearer TOKEN'
# Sample Response
{
"test_runs": [
{
"created_at": 1733770879,
"dataset_name": "big-dataset",
"description": "",
"id": "2a9dd8bd-6c29-47c8-9ca4-427f73174881",
"name": "132--after-concurrency"
},
{
"created_at": 1732651609,
"dataset_name": "Alec's Demo",
"description": "",
"id": "ab869ac0-e688-4abf-864d-cbce8dbbb1c5",
"name": "test"
},
{
"created_at": 1731966113,
"dataset_name": "big-dataset",
"description": "",
"id": "64faf9fd-b263-4477-acfc-18f576befe4d",
"name": "big-test"
}
]
}
Completion Feedback allows you to submit feedback for a specific completion in your project.
Base URL: /api/v2/projects/<project-id>/completion-feedback
Endpoint: POST /id/<completion_id>
Request Payload:
| Parameter | Type | Description | Required |
|---|---|---|---|
| feedback_attribute | str/float/int/bool | Key-value pairs of feedback attributes. | Yes |
| freeplay_feedback | str | Value must be either "positive" or "negative". | Yes |
Example cURL Request:
```bash bash curl 'https://app.freeplay.ai/api/v2/projects//completion-feedback/id/707bc301-85e9-4f02-aa97-faba8cd7774a' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer TOKEN' \ --data '{ "feedback_attribute_1": "value", "feedback_attribute_2": 10, "freeplay_feedback": "positive" }' ```Sample Response Payload:
```bash bash { "message": "Feedback created successfully" } ```Errors:
- 400 Bad Request: Returned if the request payload contains invalid data types or invalid values for
freeplay_feedback.
Trace Feedback allows you to submit feedback for a specific trace in your project.
Base URL: /api/v2/projects/<project-id>/trace-feedback
Endpoint:POST /id/<trace_id>
Request Payload:
| Parameter | Type | Description | Required |
|---|---|---|---|
| feedback_attribute | str/float/int/bool | Key-value pairs of feedback attributes. | Yes |
| freeplay_feedback | str | Value must be either "positive" or "negative". | Yes |
Example cURL Request:
```bash bash curl 'https://app.freeplay.ai/api/v2/projects//trace-feedback/id/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer TOKEN' \ --data '{ "feedback_attribute_1": "value", "feedback_attribute_2": 10, "freeplay_feedback": "positive" }' ```Sample Response Payload:
```bash bash { "message": "Feedback created successfully" } ```Errors:
- 400 Bad Request: Returned if the request payload contains invalid data types or invalid values for
freeplay_feedback.
Base URL: /api/v2/projects/<project-id>/datasets
POST/id/<dataset-id>/test-cases
** Note limit of 100 test cases per request
GET /name/<dataset-name>
{ "description": "This is a test description", "id": "4995f471-7b3f-4719-b7cc-f211a84445eb", "name": "Sample" }
</CodeGroup>
### By ID
**GET** `/id/<dataset-id>`
<CodeGroup>
```bash bash
curl --location 'https://app.freeplay.ai/api/v2/projects/8f93dd00-2eb5-4ba2-9354-86d5c6831dfd/datasets/id/4995f471-7b3f-4719-b7cc-f211a84445eb' \
--header 'Authorization: Bearer TOKEN'
{
"description": "This is a test description",
"id": "4995f471-7b3f-4719-b7cc-f211a84445eb",
"name": "Sample"
}
GET /name/<dataset-name>/test-cases
[ { "history": null, "id": "115f950c-ca48-45c3-a372-1ef84cb9da44", "output": "Bullgle", "values": { "breedA": "Beagle", "breedB": "Bulldog" } }, { "history": null, "id": "2cf1f967-8fc6-4ed7-a14e-97ffb1a3fce3", "output": "Porgi", "values": { "breedA": "Pug", "breedB": "Corgi" } }, { "history": null, "id": "8ab1d78c-7e7d-4366-ac05-963780084aa0", "output": "Greatador", "values": { "breedA": "Labrador", "breedB": "Great Dane" } } ]
</CodeGroup>
### By ID
**GET** `/id/<dataset-id>/test-cases`
<CodeGroup>
```bash bash
curl --location 'https://app.freeplay.ai/api/v2/projects/8f93dd00-2eb5-4ba2-9354-86d5c6831dfd/datasets/id/4995f471-7b3f-4719-b7cc-f211a84445eb/test-cases' \
--header 'Authorization: Bearer TOKEN'
[
{
"history": null,
"id": "115f950c-ca48-45c3-a372-1ef84cb9da44",
"output": "Bullgle",
"values": {
"breedA": "Beagle",
"breedB": "Bulldog"
}
},
{
"history": null,
"id": "2cf1f967-8fc6-4ed7-a14e-97ffb1a3fce3",
"output": "Porgi",
"values": {
"breedA": "Pug",
"breedB": "Corgi"
}
},
{
"history": null,
"id": "8ab1d78c-7e7d-4366-ac05-963780084aa0",
"output": "Greatador",
"values": {
"breedA": "Labrador",
"breedB": "Great Dane"
}
}
]
Base URL: /api/v2/projects/<project-id>/completions
Request Payload: Statistics are served in a maximum date range of 30 days
| Parameter | Type | Description | Required |
|---|---|---|---|
| to_date | str | A date that is the latest for which sessions will be returned (exclusive). Must be in the format YYYY-MM-DD. | No - Default: Today's date |
| from_date | str | A date that is the earliest for which sessions will be returned (inclusive). Must be in the format YYYY-MM-DD. | No - Default: 7 days ago |
Example errors:
- 400 Bad Request: Returned if the request payload contains invalid date ranges
POST: /statistics
{ "summary_info": { "query-rewrite": {}, "rag-directions": {}, "rag-qa": { "2025-01-10": { "auto_evals": { "Answer Accuracy": { "4": 4, "5": 12 }, "Answer Faithfulness": { "no": 1, "yes": 1 }, "Context Relevance": { "4": 5, "5": 10 } }, "client_evals": {}, "human_evals": {} }, "2025-01-11": { "auto_evals": { "Answer Accuracy": { "4": 2, "5": 12 }, "Answer Faithfulness": { "yes": 2 }, "Context Relevance": { "4": 4, "5": 12 } }, "client_evals": {}, "human_evals": {} }, "2025-01-12": { "auto_evals": { "Answer Accuracy": { "4": 2, "5": 14 }, "Answer Faithfulness": { "yes": 2 }, "Context Relevance": { "4": 2, "5": 10 } }, "client_evals": {}, "human_evals": {} }, "2025-01-13": { "auto_evals": { "Answer Accuracy": { "4": 3, "5": 10 }, "Context Relevance": { "4": 5, "5": 6 } }, "client_evals": {}, "human_evals": {} }, "2025-01-14": { "auto_evals": { "Answer Accuracy": { "5": 13 }, "Context Relevance": { "4": 1, "5": 7 } }, "client_evals": {}, "human_evals": {} } }, "rag-qa-common-elements": {}, "rag-qa-structured": {}, "rag-query": {} } }
</CodeGroup>
### By Prompt Template ID
**POST:** `/statistics/<prompt-template-id>`
<CodeGroup>
```bash bash
curl --location 'https://app.freeplay.ai/api/v2/projects/45e20940-322a-4cb8-810f-3934a56aa5e2/completions/statistics/0053c449-c736-4557-a3d4-bf3b852d0604' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer TOKEN' \
--data '{
"to_date": "2025-01-15",
"from_date": "2025-01-10"
}
'
# Sample Response
{
"summary_info": {
"rag-qa": {
"2025-01-10": {
"auto_evals": {
"Answer Accuracy": {
"4": 4,
"5": 12
},
"Answer Faithfulness": {
"no": 1,
"yes": 1
},
"Context Relevance": {
"4": 5,
"5": 10
}
},
"client_evals": {},
"human_evals": {}
},
"2025-01-11": {
"auto_evals": {
"Answer Accuracy": {
"4": 2,
"5": 12
},
"Answer Faithfulness": {
"yes": 2
},
"Context Relevance": {
"4": 4,
"5": 12
}
},
"client_evals": {},
"human_evals": {}
},
"2025-01-12": {
"auto_evals": {
"Answer Accuracy": {
"4": 2,
"5": 14
},
"Answer Faithfulness": {
"yes": 2
},
"Context Relevance": {
"4": 2,
"5": 10
}
},
"client_evals": {},
"human_evals": {}
},
"2025-01-13": {
"auto_evals": {
"Answer Accuracy": {
"4": 3,
"5": 10
},
"Context Relevance": {
"4": 5,
"5": 6
}
},
"client_evals": {},
"human_evals": {}
},
"2025-01-14": {
"auto_evals": {
"Answer Accuracy": {
"5": 13
},
"Context Relevance": {
"4": 1,
"5": 7
}
},
"client_evals": {},
"human_evals": {}
}
}
}
}
Ask AI