diff --git a/sdks/evals/quickstart.mdx b/sdks/evals/quickstart.mdx index ea921b3..6fa5e99 100644 --- a/sdks/evals/quickstart.mdx +++ b/sdks/evals/quickstart.mdx @@ -28,137 +28,188 @@ Paste this prompt into Cursor (or any coding agent) in the repo that has your ag > You are setting up PromptLayer SDK evals in this repository. -## Step 0 — Research, then optional AskQuestion +## Mandatory skill setup -### 0a — Research first +Before you inspect or edit the repository, download the PromptLayer SDK Eval Builder skill: -Quickly scan the repo for likely agent entrypoints and tool handlers (e.g. `Agent`, `Runner`, `generateText`, `tools/`, `@function_tool`, `traceTool`). Collect concrete candidates as `path:symbol`. Do not invent paths. - -### 0b — AskQuestion only if available +```bash +curl -L "https://share.promptlayer.com/api/sessions/sdk-evals/skills?format=zip" -o skills.zip +unzip -o skills.zip +``` -If the **AskQuestion** tool is in your tool list, use it once (multiple questions in one call when possible) before editing: +Read `sdk-eval-builder/SKILL.md` in full, and follow it as the primary workflow for researching the application, designing the eval, building the scaffold, running the smoke test, and analyzing results. You **must** use the downloaded skill to build the eval around the application's real agent or callable entrypoint. -1. `purpose` — What should this eval prove? - - Tool trajectory / Final-answer quality / Regression gate (tools + answer) (Recommended) -2. `entrypoint` — Which production entrypoint should be the `runner`? (researched `path:symbol` options; recommended first) -3. `tools` (`allow_multiple: true`) — Where are tools defined or executed? (researched options, or No tools / N/A) -4. `mocks` — Mock tools / side effects? No (Recommended) / Yes +If the download, extraction, or skill file is unavailable, stop and report the problem instead of inventing a replacement workflow. Do not commit `skills.zip` or the extracted skill files. -If **AskQuestion is not available**, do **not** ask these in chat and do **not** block. Proceed immediately with defaults from research: +## Repository-specific guardrail -- Purpose: regression gate if tools exist, otherwise final-answer quality -- Entrypoint: best researched candidate -- Tools: researched tool handlers, or none -- Mocks: no +Preserve the production request path. If the application uses `pl.run` for a Prompt Registry prompt, keep `pl.run`. If it calls OpenAI directly, keep the official OpenAI client. Do not migrate between them or rebuild the agent solely for the eval. + -### 0c — Brief plan, then build +### After the agent finishes -In chat (short bullets), state the purpose, entrypoint, tools, mock policy, and scorers you will use — then make the changes. Do not wait for approval unless the user already objected. +With your keys set, review the generated `evals/` file, then run it: -## Default policy (unless the user says otherwise) +```bash +promptlayer eval run ./evals +``` -- Reuse the **existing** agent entrypoint as `runner`. -- Prefer **not** mocking tools. -- Make the smallest possible instrumentation changes. -- Keep the eval file thin: import real runner + dataset + `evaluate(...)`. +## Write an eval yourself -Allowed without asking again: -- Thin adapter that only maps dataset `input` → existing function args. -- Minimal instrumentation on existing modules (`traceTool`, framework plugin, `experimental_telemetry`). +Use the **PromptLayer** tab when the prompt under test is in the Prompt Registry. Otherwise, choose the tab for the framework your production agent already uses. Do not replace `pl.run` with a direct OpenAI call, or replace an existing OpenAI call with `pl.run`, only for an eval. If you use a framework helper (OpenAI Agents, Claude, Vercel AI), tool spans are collected for you. For a custom agent with no helper, see [Runner](/sdks/evals/agent-tracing). -Forbidden unless the user explicitly approved mocks / a rewrite: -- Recreating the tool loop in `evals/`. -- Parallel “evalable slice” agents with in-memory fake tools. -- Copying production logic into a new demo agent. + + -## Goals +Use this path for agents whose prompts and tool schemas are stored in the Prompt Registry. The runner below executes each requested tool, appends its result to the agent state, and calls `pl.run` again until the model returns a final answer. -1. Use the chosen entrypoint and tool paths (do not silently switch after asking). -2. Install only packages this project needs. -3. Apply minimal instrumentation per the chosen stack and mock policy. -4. Add `evals/.eval.py` or `evals/.eval.ts` with a **10-case** dataset and `evaluate(...)`. -5. Make it runnable with `promptlayer eval run `. + +```bash Python +pip install promptlayer +``` -## Product constraints (do not invent APIs) +```bash JavaScript +npm install promptlayer +``` + -Follow current PromptLayer SDK evals docs: +This example evaluates the `production` release of a Prompt Registry prompt named `weather-agent`. Configure it with: -- Primary API: top-level `evaluate(...)` (Python also has `aevaluate` for async runners). -- Recommended run path: put `evaluate(...)` in a file, then run `promptlayer eval run `. -- Do **not** use a removed/demoted `client.evals.run` API. -- The runner can be **any** existing agent entrypoint (any framework or a plain function). Do not rebuild the agent to match an example stack. -- For Trajectory, tools must appear as Trace spans named `Tool: `. Prefer the lightest path that already fits the repo: - - OpenAI Agents (if used): `instrument_openai_agents()` / `instrumentOpenAIAgents()` from `promptlayer.integrations.openai_agents` / `promptlayer/openai-agents`. No `traceTool` needed. - - Claude Agents (if used): call `get_claude_config()` / `getClaudeConfig()` **inside** the eval runner and pass `plugin` + `env` into Claude Agent options. No `traceTool` needed. - - Vercel AI SDK (JS, if used): enable `experimental_telemetry` on the existing `generateText` / agent calls. No separate NodeSDK needed for the eval path. - - LangChain / LangSmith (if used): LangSmith OTEL bridge → `https://api.promptlayer.com/v1/traces` (see Integrations). - - Pydantic AI (if used): `logfire.instrument_pydantic_ai()` with OTLP export to PromptLayer `/v1/traces`. - - OpenClaw (if used): `@promptlayer/openclaw-promptlayer` plugin enabled; reuse the real OpenClaw entrypoint as `runner`. - - LiteLLM (Python, if used): `PromptLayer(enable_tracing=True)`, optional `litellm.success_callback = ["promptlayer"]`, and `traceTool` on **existing** tool handlers. - - Anything else / custom agent: wrap **existing** handlers with PromptLayer `traceTool` (JS: `pl.traceTool("name", fn)`; Python: `@pl.traceTool(name="name")`). -- For custom / raw OpenAI client runners: keep the repo’s existing client. Prefer the official OpenAI client over `pl.openai.OpenAI()` / `new pl.OpenAI()` unless the repo already depends on the PromptLayer provider wrapper. -- Dataset cases use `input`, plus optional `expected` (expected output). Add `expected_trace` / `expectedTrace` only when using Trajectory. Extra keys become custom TEXT columns with that exact title (no reserved names: `input`, `expected`, `Output`, `Trace`, `expected_trace` / `expectedTrace`, or legacy `Input` / `Expected` / `Expected Trace`). -- Builtin sheet columns are `input`, `expected`, `expected_trace` / `expectedTrace`, `Output`, and `Trace`. Scorers take `source_column` / `sourceColumn`. Compare / Contains / Trajectory use `expected` (literal or inline scenarios) or `expected_column` / `expectedColumn` (column-backed). -- Prefer scorers that match the purpose: Trajectory for tool path, Contains / Compare / LLM assertion for the final answer. -- Results must land in PromptLayer Tables via the SDK eval runner. +- A `user_message` input variable +- An `agent_messages` [message placeholder](/features/prompt-registry/placeholder-messages) after the user message +- The instruction: `Use get_weather, then answer in one short sentence.` +- A schema-only Registry tool named `get_weather` with a required string parameter named `city` -## Step 1 — Confirm from research + +```python Python +# evals/weather_agent.eval.py +import json -- Confirm the production agent entrypoint and tool execution path. -- Inventory the real input shape. -- Note existing PromptLayer / OpenTelemetry / provider instrumentation. +from promptlayer import ( + PromptLayer, + contains_scorer, + evaluate, + trajectory_scorer, +) -## Step 2 — Install +pl = PromptLayer() -- Only install what is missing for the chosen stack. -- Document required env vars: `PROMPTLAYER_API_KEY`, plus whatever the real agent already needs. +@pl.traceTool(name="get_weather") +def get_weather(city: str) -> dict: + return {"city": city, "temperature_f": 72, "conditions": "sunny"} -## Step 3 — Instrumentation / mocks +def run_agent(user_message: str) -> str: + agent_messages = [] -- **Mocks = no:** edit existing tool handlers / agent setup in place only. -- **Mocks = yes:** mock only approved side effects; keep the real loop/entrypoint; do not rebuild the agent. -- Use a framework helper only when the repo already uses that stack; otherwise use `traceTool` (or existing OTEL) on real tool handlers. -- Do not break production paths; keep diffs small and eval-focused. + for _ in range(5): + response = pl.run( + prompt_name="weather-agent", + prompt_release_label="production", + input_variables={ + "user_message": user_message, + "agent_messages": agent_messages, + }, + ) + message = response["prompt_blueprint"]["prompt_template"]["messages"][-1] -## Step 4 — Thin eval file + if not message.get("tool_calls"): + return message["content"][-1]["text"] -Create `evals/.eval.py` or `evals/.eval.ts` that: + agent_messages.append(message) + for tool_call in message["tool_calls"]: + if tool_call["function"]["name"] != "get_weather": + raise ValueError(f"Unknown tool: {tool_call['function']['name']}") -1. Imports `evaluate` / scorers. -2. Imports the real agent entrypoint (or a tiny input adapter). -3. Avoids redefining the agent/tool loop unless the user approved that approach. -4. Builds a **dataset of 10 cases** aligned to the purpose and input shape. -5. Chooses scorers from the purpose (Trajectory when tools matter; Contains / Compare / LLM assertion for answers). Default `passing_score` / `passingScore` to `0.8`. -6. Calls top-level `evaluate("", ...)`. + arguments = json.loads(tool_call["function"]["arguments"]) + result = get_weather(**arguments) + agent_messages.append({ + "role": "tool", + "tool_call_id": tool_call["id"], + "content": [{"type": "text", "text": json.dumps(result)}], + }) -## Step 5 — Verify + raise RuntimeError("Agent exceeded the five-turn limit") -- Show how to run: `promptlayer eval run ./evals` -- Quote the import of the real runner. -- Summarize purpose, entrypoint, tools, and mock policy used. -- List files changed, 10 dataset inputs, env vars, and TODOs. +evaluate( + "weather-agent-eval", + dataset=[{"input": "What is the weather in Tokyo?"}], + runner=run_agent, + scorers=[ + trajectory_scorer(expected=[["get_weather"]], mode="non_strict"), + contains_scorer(source_column="Output", expected="72"), + ], + passing_score=1.0, +) +``` -## Output format +```javascript JavaScript +// evals/weather_agent.eval.ts +import { + PromptLayer, + containsScorer, + evaluate, + trajectoryScorer, +} from "promptlayer"; + +const pl = new PromptLayer(); + +const getWeather = pl.traceTool( + "get_weather", + async ({ city }) => ({ city, temperature_f: 72, conditions: "sunny" }) +); -- Research → AskQuestion **only if available** → otherwise proceed with defaults → build. -- Never dump a chat quiz when AskQuestion is missing. -- End with a short checklist: installs, files added/changed, runner import path, how to run, env vars, remaining TODOs. -- Do not silently invent a replacement agent. - +async function runAgent(userMessage) { + const agentMessages = []; + + for (let turn = 0; turn < 5; turn++) { + const response = await pl.run({ + promptName: "weather-agent", + promptReleaseLabel: "production", + inputVariables: { + user_message: userMessage, + agent_messages: agentMessages, + }, + }); + const messages = response.prompt_blueprint.prompt_template.messages; + const message = messages.at(-1); + + if (!message.tool_calls?.length) { + return message.content.at(-1).text; + } -### After the agent finishes + agentMessages.push(message); + for (const toolCall of message.tool_calls) { + if (toolCall.function.name !== "get_weather") { + throw new Error(`Unknown tool: ${toolCall.function.name}`); + } + + const result = await getWeather(JSON.parse(toolCall.function.arguments)); + agentMessages.push({ + role: "tool", + tool_call_id: toolCall.id, + content: [{ type: "text", text: JSON.stringify(result) }], + }); + } + } -With your keys set, review the generated `evals/` file, then run it: + throw new Error("Agent exceeded the five-turn limit"); +} -```bash -promptlayer eval run ./evals +await evaluate("weather-agent-eval", { + dataset: [{ input: "What is the weather in Tokyo?" }], + runner: runAgent, + scorers: [ + trajectoryScorer({ expected: [["get_weather"]], mode: "non_strict" }), + containsScorer({ sourceColumn: "Output", expected: "72" }), + ], + passingScore: 1.0, +}); ``` + -## Write an eval yourself - -Pick the tab that matches your stack. If you use a framework helper (OpenAI Agents, Claude, Vercel AI), tool spans are collected for you. For a custom agent with no helper, see [Runner](/sdks/evals/agent-tracing). +Use `promptVersion` / `prompt_version` instead when you need to pin an exact version. Keep the release label when the eval should continuously test the prompt currently assigned to an environment. + - @@ -443,7 +494,7 @@ Install: pip install promptlayer litellm ``` -Enable PromptLayer tracing with `PromptLayer(enable_tracing=True)`, wrap tools with [`traceTool`](/features/observability/traces/manual-tracing#trace-tools) for Trajectory, and set LiteLLM's PromptLayer callback for request logging. LiteLLM does not emit separate `Tool:` spans on its own — `traceTool` is a no-op unless that `PromptLayer` instance has tracing enabled. +Wrap tools with [`traceTool`](/features/observability/traces/manual-tracing#trace-tools) for Trajectory, and set LiteLLM's PromptLayer callback for request logging. LiteLLM does not emit separate `Tool:` spans on its own; during an eval, `evaluate(...)` supplies the active tracer used by `traceTool`. ```python # evals/litellm_weather.eval.py @@ -452,8 +503,7 @@ import litellm from litellm import completion from promptlayer import PromptLayer, evaluate, contains_scorer, trajectory_scorer -# Required: PromptLayer tracing for Tool: spans / Trajectory -pl = PromptLayer(enable_tracing=True) +pl = PromptLayer() # Optional: also log LiteLLM LLM requests to PromptLayer litellm.success_callback = ["promptlayer"] @@ -522,7 +572,7 @@ evaluate( ) ``` -`evaluate(...)` enables tracing on its own client for nesting under each eval case, but your separate `PromptLayer(...)` instance still needs `enable_tracing=True` for `@pl.traceTool`. Callbacks: [LiteLLM](/features/observability/traces/integrations#litellm) and the [LiteLLM PromptLayer docs](https://docs.litellm.ai/docs/observability/promptlayer_integration). +Set `enable_tracing=True` only if the same `PromptLayer` client and traced tools also run outside `evaluate(...)`. Callbacks: [LiteLLM](/features/observability/traces/integrations#litellm) and the [LiteLLM PromptLayer docs](https://docs.litellm.ai/docs/observability/promptlayer_integration).