[docs] Create evals guide

[onmax]

Description

Following the work at nuxt/nuxt.com#2099 and mattpocock/evalite#339,

I think it would be beneficial to add a guide on how to create evals, including recommendations and best practices.

Please let me know if you think this is a good idea or if you think it should not be included in the toolkit documentation.

Proposed Solution

I am happy to work on this! 🙌

[HugoRCD]

@onmax It's clearly one of the things I had planned, so I'm very happy to be collaborating on it! The aim of the module is to be a toolkit, so it should explain and improve the DX of everything to do with MCP!

[onmax]

Yes, I've been thinking about how we could improve the DX and add evals out of the box in the toolkit, but I'm not sure how to go about it.

That's why I was thinking of providing just a written guide with 'Evalite' as an example.

Following Nuxt's philosophy, I also thought that we shouldn't force users to use 'Evalite' and that we should make it 'Eval-library-agnostic'. But maybe I'm overcomplicating things.

If you have any other ideas or approaches, please share them!

[HugoRCD]

I haven't experimented enough with evals apart from Evalite and I don't think I know the subject as well as you do at the moment. I agree with respecting Nuxt's philosophy and not forcing Evalite, but if I'm not mistaken there aren't really any other reliable solutions at the moment and I'm mostly of the opinion to use the best tool for my use case so I'll probably say to all-in on Evalite for now 🤔

[onmax]

I am also completely new to this... so please don't take my input that serious haha.

But ok, i do agree. Let's focus on evalite.

What do you think about this API:

// server/mcp/tools/bmi.ts
export default defineMcpTool({
  name: 'calculate_bmi',
  description: 'Calculate BMI from weight and height',
  inputSchema: { weightKg: z.number(), heightM: z.number() },
  handler: async ({ weightKg, heightM }) => ({
    content: [{ type: 'text', text: `BMI: ${weightKg / (heightM ** 2)}` }]
  }),
  // NEW: Inline evals
  evals: [
    { 
      input: 'Calculate BMI for 70kg and 1.75m', 
      expected: { weightKg: 70, heightM: 1.75 } 
    }
  ]
})

Multi-step tools

// server/mcp/evals/recipe.eval.ts
import { defineMcpEval } from '@nuxtjs/mcp-toolkit-evals'

export default defineMcpEval({
  name: 'Recipe Workflow',
  maxSteps: 3,
  data: [
    {
      input: 'Find chicken recipes and show me the first one',
      expected: [
        { toolName: 'search_recipes', input: { query: 'chicken' } },
        { toolName: 'get_recipe' } 
      ]
    }
  ]
})

---

config would look like

// nuxt.config.ts
import { defineNuxtConfig } from 'nuxt'
import { openai } from '@ai-sdk/openai'

export default defineNuxtConfig({
  modules: [
    '@nuxtjs/mcp-toolkit',
  ],

  mcp: {
	// other config

    evalite: { // or evals?
      model: openai('gpt-5', { 
        // Note: The AI SDK typically finds the API key via process.env
      }),
      mcpUrl: process.env.MCP_EVAL_URL, 
      // maxSteps: 3, // other configs
    }
  }
})

[HugoRCD]

@onmax I'm not sure I really like testing at the same level as definition 🤔 On the other hand, we can document the process for testing on our MCP with the ai-gateway! nuxt/nuxt.com@0d1a522

[atinux]

Love the idea of having a guide for evals!

Having evals within defineMCPTool sounds good, but will make the production bundle bigger ⚠️

I rather have another .eval.ts file alongside the tools definitions.

At first I recommend to only start with documenting it, abstracting on a beta library is maybe too early.

[HugoRCD]

@onmax Have you started working on this or not? absolutely nothing pressing but I thought I'd get on it within the week just in case, otherwise I'll be waiting 😁

[onmax]

Yes. I started working on this. But I was on time off the last few days. Expect a PR today or tomorrow :)