Initialize repository with benchmark template

.github/workflows/deploy.yml

@@ -0,0 +1,69 @@
+name: Deploy Next.js site to Pages
+
+on:
+  push:
+    branches: ["main"]
+  pull_request:
+    types: [opened, reopened, synchronize, closed]
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pull-requests: write
+  pages: write
+  id-token: write
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.ref_name }}
+  cancel-in-progress: true
+
+jobs:
+  build-and-deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Install Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: 22
+
+      - name: Setup Bun
+        uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: latest
+
+      - name: Install dependencies
+        run: bun install --frozen-lockfile
+        working-directory: site
+
+      - name: Setup Pages
+        id: setup_pages
+        uses: actions/configure-pages@v5
+
+      - name: Set base uri
+        run: |
+          if [ "${{ github.event_name }}" = "pull_request" ]; then
+            echo "NEXT_PUBLIC_BASE_PATH=${{ steps.setup_pages.outputs.base_path }}/pr-preview/pr-${{ github.event.pull_request.number }}" >> "$GITHUB_ENV"
+          else
+            echo "NEXT_PUBLIC_BASE_PATH=${{ steps.setup_pages.outputs.base_path }}" >> "$GITHUB_ENV"
+          fi
+
+      - name: Build with Next.js
+        run: bun run build
+        working-directory: site
+
+      - name: Deploy preview
+        if: github.event_name == 'pull_request'
+        uses: rossjrw/pr-preview-action@v1
+        with:
+          source-dir: site/out
+
+      - name: Deploy production
+        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+        uses: JamesIves/github-pages-deploy-action@v4
+        with:
+          clean-exclude: pr-preview/
+          force: false
+          folder: site/out

.gitignore

@@ -0,0 +1,49 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+node_modules
+.pnp
+.pnp.*
+.yarn/*
+!.yarn/patches
+!.yarn/plugins
+!.yarn/releases
+!.yarn/versions
+
+__pycache__/
+*.py[cod]
+*$py.class
+
+# testing
+coverage
+
+# next.js
+.next/
+
+# The `out` directory should not be ignored by version control
+out/
+
+# production
+build
+
+# misc
+.DS_Store
+*.pem
+*~
+\#*
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# local env files
+.env*.local
+.env*
+
+# vercel
+.vercel
+
+# typescript
+*.tsbuildinfo
+next-env.d.ts

.zealt/config.json

@@ -0,0 +1,5 @@
+{
+  "title": "TanStack Benchmark",
+  "description": "Performance results of AI coding models on TanStack tasks, measuring success rate and execution time with high precision.",
+  "github_repo": "https://github.com/kweizh/tanstack-benchmark"
+}

README.md

@@ -1 +1,46 @@
-# tanstack-benchmark
+
+# TanStack Benchmark
+
+This repository contains benchmarks for evaluating AI models on **TanStack**.
+
+You can view the evaluation reports at [https://kweizh.github.io/tanstack-benchmark/](https://kweizh.github.io/tanstack-benchmark/).
+
+## Project Structure
+
+- `tasks/`: Contains the benchmark tasks, each with its own instructions.
+- `jobs/`: Stores the results of benchmark runs.
+- `site/`: A Next.js application to visualize benchmark results.
+
+## Getting Started
+
+This benchmark is evaluated using the [Harbor framework](https://github.com/harbor-framework/harbor) and the [Pochi agent](https://github.com/TabbyML/pochi).
+
+### Running Evaluation
+
+You can run the evaluation using the Harbor CLI. Here is an example:
+
+```bash
+harbor run \
+  --agent codex \
+  --model "gpt-5.2-codex" \
+  --env daytona \
+  --path ./tasks \
+  --n-attempts 1 \
+  --max-retries 5 \
+  --n-concurrent 5 \
+  --retry-include RuntimeError \
+  --retry-include DaytonaError \
+  --retry-include AgentTimeoutError
+```
+
+### Evaluation Details
+
+Before starting the evaluation, you should set the necessary environment variables for your chosen agent.
+For example, if using Pochi, you should export `POCHI_API_KEY`.
+
+Evaluation can be run locally with Docker (default), or using [Daytona.io](https://www.daytona.io/) by setting `--env daytona`.
+
+When running with Daytona, please note that Daytona blocks some network access for tier 1 and tier 2 users. If you meet any network issues, please refer to [Daytona network limits](https://www.daytona.io/docs/en/network-limits/).
+
+---
+Generated by [Zealt](https://github.com/TabbyML/zealt)

plan.md

@@ -0,0 +1,136 @@
+# TanStack Benchmark Research Report
+
+## 1. Library Overview
+
+**Description**: TanStack is a collection of high-quality, open-source headless libraries for web development. It focuses on the "hard parts" of application development: state management, routing, data grids, and forms. The ecosystem's flagship is **TanStack Start**, a full-stack React framework that integrates these libraries into a cohesive, type-safe development experience.
+
+**Ecosystem Role**: TanStack provides the foundational "engine" for modern web apps. Unlike opinionated UI kits, TanStack libraries are "headless," providing logic and state without markup, allowing developers to use any UI library (e.g., Tailwind, Shadcn UI) while maintaining strict type safety from the database to the browser.
+
+**Project Setup**:
+The recommended way to initialize a full-stack project is via the TanStack CLI:
+```bash
+npx @tanstack/cli@latest create
+```
+Standard project structure for TanStack Start:
+- `app/routes/`: File-based routing directory.
+- `app/routeTree.gen.ts`: Automatically generated type-safe route tree.
+- `app/ssr.tsx` & `app/client.tsx`: Entry points for server and client.
+- `app/router.tsx`: Shared router configuration.
+
+---
+
+## 2. Core Primitives & APIs
+
+### TanStack Query (Server State)
+- **Concept**: Manages asynchronous state (fetching, caching, synchronization).
+- **Core APIs**: `useQuery`, `useMutation`, `queryOptions`.
+- **Code Snippet**:
+```typescript
+const postsQuery = queryOptions({
+  queryKey: ['posts'],
+  queryFn: () => fetch('/api/posts').then(r => r.json()),
+})
+
+function Posts() {
+  const { data } = useQuery(postsQuery)
+  return <ul>{data?.map(post => <li key={post.id}>{post.title}</li>)}</ul>
+}
+```
+- **Docs**: [TanStack Query Reference](https://tanstack.com/query/latest/docs/framework/react/reference/useQuery)
+
+### TanStack Router (Type-Safe Routing)
+- **Concept**: File-based routing with 100% type safety for paths, params, and search state.
+- **Core APIs**: `createFileRoute`, `Link`, `useLoaderData`.
+- **Code Snippet**:
+```typescript
+// routes/posts.$postId.tsx
+export const Route = createFileRoute('/posts/$postId')({
+  loader: ({ params }) => fetchPost(params.postId),
+  component: PostComponent,
+})
+
+function PostComponent() {
+  const data = Route.useLoaderData()
+  return <div>{data.title}</div>
+}
+```
+- **Docs**: [TanStack Router Guide](https://tanstack.com/router/latest/docs/routing/file-based-routing)
+
+### TanStack Table (Headless Data Grid)
+- **Concept**: Logic engine for complex tables (sorting, filtering, pagination).
+- **Core APIs**: `useReactTable`, `createColumnHelper`.
+- **Code Snippet**:
+```typescript
+const table = useReactTable({
+  data,
+  columns,
+  getCoreRowModel: getCoreRowModel(),
+  getPaginationRowModel: getPaginationRowModel(),
+})
+
+// Render using table.getHeaderGroups() and table.getRowModel().rows
+```
+- **Docs**: [TanStack Table Core APIs](https://tanstack.com/table/latest/docs/api/core/table)
+
+### TanStack Start (Full-Stack)
+- **Concept**: SSR, Streaming, and Server Functions (RPCs).
+- **Core APIs**: `createServerFn`, `createFileRoute`.
+- **Code Snippet**:
+```typescript
+const updateCount = createServerFn({ method: 'POST' })
+  .validator((d: number) => d)
+  .handler(async ({ data }) => {
+    // Server-side logic (DB update, etc.)
+    return { success: true }
+  })
+```
+- **Docs**: [TanStack Start Overview](https://tanstack.com/start/latest/docs/framework/react/overview)
+
+---
+
+## 3. Real-World Use Cases & Templates
+
+- **SaaS Admin Dashboards**: Combining **Table** for data grids, **Query** for server state, and **Router** for deeply nested layouts and URL-driven filters.
+- **E-commerce Product Filters**: Using **Router's Search Param Validation** (Zod-integrated) to manage complex filtering states in the URL.
+- **AI-Powered Chat Apps**: Using **TanStack AI** for streaming responses and tool-calling with approval workflows.
+- **Template**: [Trellaux](https://github.com/TanStack/router/tree/main/examples/react/start-trellaux) - A full-stack Trello clone showing Start, Query, and complex drag-and-drop state.
+
+---
+
+## 4. Developer Friction Points
+
+1.  **Router Type Generation**: The `routeTree.gen.ts` file must be generated via a background watcher. AI agents often struggle to trigger this generation or understand that the file is missing during initial setup.
+2.  **SSR Hydration with Query**: Misconfiguring `staleTime` or `gcTime` during SSR can lead to "Hydration Mismatch" errors where the server and client data differ.
+3.  **Table Column Typing**: Defining complex columns with custom cell renderers and meta-data requires deep understanding of TypeScript generics, often leading to "Type instantiation is excessively deep" errors.
+4.  **Headless Complexity**: The "Headless" nature means no default UI. Implementing a basic accessible table or form requires significant boilerplate (e.g., mapping over header groups).
+
+---
+
+## 5. Evaluation Ideas
+
+### Simple
+- Create a type-safe navigation menu with active link highlighting using TanStack Router.
+- Implement a basic "Todo" list that fetches and creates items using TanStack Query.
+
+### Intermediate
+- Build a paginated data table with TanStack Table including server-side sorting.
+- Create a multi-step registration form with TanStack Form and Zod validation.
+- Implement a "Search" page where all filters (query, category, price range) are synced to the URL via TanStack Router.
+
+### Complex
+- Build a full-stack "Counter" app in TanStack Start using Server Functions and SQLite.
+- Implement an "Optimistic Update" flow for a nested comment system using TanStack Query.
+- Create an AI Chat interface using TanStack AI that includes a "Tool Approval" step for database writes.
+
+---
+
+## 6. Sources
+
+1. [TanStack Official Site](https://tanstack.com/) - Main ecosystem hub.
+2. [TanStack Query Docs](https://tanstack.com/query/latest/docs) - Server-state management reference.
+3. [TanStack Router Docs](https://tanstack.com/router/latest/docs) - Type-safe routing and URL state reference.
+4. [TanStack Start Docs](https://tanstack.com/start/latest/docs) - Full-stack framework and SSR reference.
+5. [TanStack Table Docs](https://tanstack.com/table/latest/docs) - Headless table engine reference.
+6. [TanStack Form Docs](https://tanstack.com/form/latest/docs) - Headless form state reference.
+7. [TanStack AI Overview](https://tanstack.com/ai/latest/docs/getting-started/overview) - AI SDK and tool calling reference.
+8. [TanStack GitHub Repository](https://github.com/TanStack) - Source code and community examples.

scratchpad/pending-tasks/headless_table_initialization_and_typing/instruction.md

@@ -0,0 +1,8 @@
+TanStack Table is a headless logic engine, requiring strict typing for complex columns and manual boilerplate mapping to render the UI grid without "excessively deep" type instantiation errors.
+
+You need to instantiate a data table using `useReactTable` and `createColumnHelper` for a specific `Employee` interface, and build the standard HTML table markup to display the data. 
+
+**Constraints:**
+- Must render the standard HTML `<table>`, `<thead>`, and `<tbody>` structure by manually mapping over `table.getHeaderGroups()` and `table.getRowModel().rows`.
+- Must create at least one custom cell renderer (e.g., formatting a date or rendering an Action button).
+- Do NOT use any pre-built UI library table components (like MUI DataGrid or AG Grid).

scratchpad/pending-tasks/optimistic_updates_with_tanstack_query/instruction.md

@@ -0,0 +1,8 @@
+TanStack Query manages asynchronous server state and allows optimistic updates for a snappy user experience during data mutations.
+
+You need to implement a `useMutation` hook for adding a new "Post" that optimistically updates the local cache before the network request finishes, and rolls back if the network request fails. 
+
+**Constraints:**
+- Must synchronously update the cache array for the `['posts']` query key inside the `onMutate` callback.
+- Must implement the rollback logic in the `onError` callback using the context returned from `onMutate`.
+- Must trigger a background refetch via `onSettled` to ensure synchronization.

scratchpad/pending-tasks/resolving_query_ssr_hydration_mismatch/instruction.md

@@ -0,0 +1,8 @@
+When combining TanStack Query with Server-Side Rendering (SSR), misconfiguring caching durations often leads to "Hydration Mismatch" errors where server HTML and initial client state differ.
+
+You need to configure the global `QueryClient` initialization in a TanStack Start `app.tsx` file to properly align server and client caching behavior and prevent immediate refetching on hydration. 
+
+**Constraints:**
+- Must set a default `staleTime` strictly greater than `0` (e.g., `60 * 1000`) in the default options to prevent instant invalidation.
+- Must conditionally initialize the `QueryClient` so it is not shared across users during SSR, but remains a singleton on the client.
+- Do NOT alter any specific component's `useQuery` configurations; apply the fix at the root provider level.

scratchpad/pending-tasks/server_function_creation_in_tanstack_start/instruction.md

@@ -0,0 +1,8 @@
+TanStack Start supports full-stack RPCs via Server Functions, allowing secure server-side logic and database operations to be called directly from client components.
+
+You need to create a `updateCount` server function in a TanStack Start application using `createServerFn` that accepts an increment amount, executes dummy server-side logic, and returns a success payload. 
+
+**Constraints:**
+- Must use the `.validator()` method to ensure the payload type is strictly an integer.
+- Must configure the server function method strictly as `POST`.
+- Do NOT write directly to a real database; use a mock asynchronous return.

scratchpad/pending-tasks/synchronized_url_search_parameters/instruction.md

@@ -0,0 +1,8 @@
+TanStack Router allows deep integration with validation libraries like Zod to manage complex, type-safe filtering states directly in the URL search params.
+
+You need to create a file route definition for `/products` that strictly types and validates URL search parameters (`category` as a string, `inStock` as a boolean) using the `validateSearch` option. 
+
+**Constraints:**
+- Must use Zod (`z.object`) to validate the search parameters.
+- Must provide fallback default values (`category` defaults to `"all"`, `inStock` defaults to `true`) if the parameters are omitted from the URL.
+- Do NOT generate the `routeTree.gen.ts` file manually.

scratchpad/pending-tasks/type_safe_navigation_menu/instruction.md

@@ -0,0 +1,7 @@
+TanStack Router provides file-based routing and 100% type safety for paths, ensuring broken links are caught at compile time. 
+
+You need to implement a navigation menu component `Nav.tsx` utilizing TanStack Router's `<Link>` component to navigate between `/`, `/posts`, and `/settings` in a standard React environment. 
+
+**Constraints:**
+- Must apply a specific CSS class `active-link` to the currently active route using the `activeProps` API.
+- Do NOT use `react-router-dom` or standard HTML `<a>` tags for internal routing.