Initialize repository with benchmark template

.github/workflows/deploy.yml

@@ -0,0 +1,69 @@
+name: Deploy Next.js site to Pages
+
+on:
+  push:
+    branches: ["main"]
+  pull_request:
+    types: [opened, reopened, synchronize, closed]
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pull-requests: write
+  pages: write
+  id-token: write
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.ref_name }}
+  cancel-in-progress: true
+
+jobs:
+  build-and-deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Install Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: 22
+
+      - name: Setup Bun
+        uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: latest
+
+      - name: Install dependencies
+        run: bun install --frozen-lockfile
+        working-directory: site
+
+      - name: Setup Pages
+        id: setup_pages
+        uses: actions/configure-pages@v5
+
+      - name: Set base uri
+        run: |
+          if [ "${{ github.event_name }}" = "pull_request" ]; then
+            echo "NEXT_PUBLIC_BASE_PATH=${{ steps.setup_pages.outputs.base_path }}/pr-preview/pr-${{ github.event.pull_request.number }}" >> "$GITHUB_ENV"
+          else
+            echo "NEXT_PUBLIC_BASE_PATH=${{ steps.setup_pages.outputs.base_path }}" >> "$GITHUB_ENV"
+          fi
+
+      - name: Build with Next.js
+        run: bun run build
+        working-directory: site
+
+      - name: Deploy preview
+        if: github.event_name == 'pull_request'
+        uses: rossjrw/pr-preview-action@v1
+        with:
+          source-dir: site/out
+
+      - name: Deploy production
+        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+        uses: JamesIves/github-pages-deploy-action@v4
+        with:
+          clean-exclude: pr-preview/
+          force: false
+          folder: site/out

.gitignore

@@ -0,0 +1,49 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+node_modules
+.pnp
+.pnp.*
+.yarn/*
+!.yarn/patches
+!.yarn/plugins
+!.yarn/releases
+!.yarn/versions
+
+__pycache__/
+*.py[cod]
+*$py.class
+
+# testing
+coverage
+
+# next.js
+.next/
+
+# The `out` directory should not be ignored by version control
+out/
+
+# production
+build
+
+# misc
+.DS_Store
+*.pem
+*~
+\#*
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# local env files
+.env*.local
+.env*
+
+# vercel
+.vercel
+
+# typescript
+*.tsbuildinfo
+next-env.d.ts

.zealt/config.json

@@ -0,0 +1,5 @@
+{
+  "title": "Godot Benchmark",
+  "description": "Performance results of AI coding models on Godot tasks, measuring success rate and execution time with high precision.",
+  "github_repo": "https://github.com/kweizh/godot-benchmark"
+}

README.md

@@ -1 +1,46 @@
-# godot-benchmark
+
+# Godot Benchmark
+
+This repository contains benchmarks for evaluating AI models on **Godot**.
+
+You can view the evaluation reports at [https://kweizh.github.io/godot-benchmark/](https://kweizh.github.io/godot-benchmark/).
+
+## Project Structure
+
+- `tasks/`: Contains the benchmark tasks, each with its own instructions.
+- `jobs/`: Stores the results of benchmark runs.
+- `site/`: A Next.js application to visualize benchmark results.
+
+## Getting Started
+
+This benchmark is evaluated using the [Harbor framework](https://github.com/harbor-framework/harbor) and the [Pochi agent](https://github.com/TabbyML/pochi).
+
+### Running Evaluation
+
+You can run the evaluation using the Harbor CLI. Here is an example:
+
+```bash
+harbor run \
+  --agent codex \
+  --model "gpt-5.2-codex" \
+  --env daytona \
+  --path ./tasks \
+  --n-attempts 1 \
+  --max-retries 5 \
+  --n-concurrent 5 \
+  --retry-include RuntimeError \
+  --retry-include DaytonaError \
+  --retry-include AgentTimeoutError
+```
+
+### Evaluation Details
+
+Before starting the evaluation, you should set the necessary environment variables for your chosen agent.
+For example, if using Pochi, you should export `POCHI_API_KEY`.
+
+Evaluation can be run locally with Docker (default), or using [Daytona.io](https://www.daytona.io/) by setting `--env daytona`.
+
+When running with Daytona, please note that Daytona blocks some network access for tier 1 and tier 2 users. If you meet any network issues, please refer to [Daytona network limits](https://www.daytona.io/docs/en/network-limits/).
+
+---
+Generated by [Zealt](https://github.com/TabbyML/zealt)

plan.md

@@ -0,0 +1,108 @@
+# Godot Engine Evaluation Dataset Research
+
+### 1. Library Overview
+*   **Description**: Godot Engine is a free, open-source, cross-platform 2D and 3D game engine. It features a unique node-and-scene architecture, a dedicated Python-like scripting language (GDScript), and support for C# and C++ (via GDExtension).
+*   **Ecosystem Role**: A major competitor to Unity and Unreal Engine, favored for its lightweight nature, permissive MIT license, and excellent 2D capabilities. It is increasingly used for 3D games and non-game applications (tools, simulators).
+*   **Project Setup**:
+    1.  **Download**: Godot is a single executable. No installation required.
+    2.  **Initialize**: Create a new folder and a `project.godot` file (automatically done via the Project Manager).
+    3.  **CLI**: 
+        *   Open editor: `godot -e`
+        *   Run project: `godot`
+        *   Export: `godot --export-release "Linux/X11" path/to/export`
+    4.  **Structure**: Standard practice uses `res://` as the root. Common folders: `scenes/`, `scripts/`, `assets/`, `prefabs/`.
+
+### 2. Core Primitives & APIs
+
+*   **Nodes & Scenes**: Everything is a Node. Nodes are organized into Scenes. Scenes can be instanced within other scenes.
+    *   [Nodes and Scenes Docs](https://docs.godotengine.org/en/stable/getting_started/step_by_step/nodes_and_scenes.html)
+*   **GDScript**: A high-level, dynamically typed language optimized for Godot.
+    *   [GDScript Basics](https://docs.godotengine.org/en/stable/tutorials/scripting/gdscript/gdscript_basics.html)
+    *   **Snippet (Basic Player)**:
+        ```gdscript
+        extends CharacterBody2D
+
+        @export var speed = 300.0
+        @export var jump_velocity = -400.0
+
+        func _physics_process(delta):
+            # Add gravity
+            if not is_on_floor():
+                velocity += get_gravity() * delta
+
+            # Handle Jump
+            if Input.is_action_just_pressed("ui_accept") and is_on_floor():
+                velocity.y = jump_velocity
+
+            # Get input direction
+            var direction = Input.get_axis("ui_left", "ui_right")
+            if direction:
+                velocity.x = direction * speed
+            else:
+                velocity.x = move_toward(velocity.x, 0, speed)
+
+            move_and_slide()
+        ```
+*   **Signals**: The Observer pattern implementation. Used for decoupled communication.
+    *   [Using Signals](https://docs.godotengine.org/en/stable/getting_started/step_by_step/signals.html)
+    *   **Snippet (Connecting via Code)**:
+        ```gdscript
+        func _ready():
+            var timer = get_node("Timer")
+            timer.timeout.connect(_on_timer_timeout)
+
+        func _on_timer_timeout():
+            print("Timer finished!")
+        ```
+*   **Resources**: Data containers (e.g., Textures, Scripts, custom data).
+    *   [Resources Docs](https://docs.godotengine.org/en/stable/tutorials/scripting/resources.html)
+    *   **Snippet (Custom Resource)**:
+        ```gdscript
+        # item_data.gd
+        extends Resource
+        class_name ItemData
+
+        @export var name: String
+        @export var icon: Texture2D
+        @export var damage: int
+        ```
+*   **Networking**: High-level multiplayer API using RPCs and Synchronizers.
+    *   [Multiplayer Docs](https://docs.godotengine.org/en/stable/tutorials/networking/high_level_multiplayer.html)
+    *   **Snippet (RPC)**:
+        ```gdscript
+        @rpc("any_peer", "call_local")
+        func update_score(value):
+            score += value
+        ```
+*   **GDExtension (C++)**: High-performance extension system without recompiling the engine.
+    *   [GDExtension Docs](https://docs.godotengine.org/en/stable/tutorials/scripting/gdextension/gdextension_cpp_example.html)
+    *   **Key Concept**: Requires a `.gdextension` config file to map shared libraries to platforms.
+
+### 3. Real-World Use Cases & Templates
+*   **SaaS/Tool UI**: Using `Control` nodes, `GridContainer`, and `Theme` for complex editors.
+*   **Multiplayer FPS/Platformer**: Utilizing `MultiplayerSynchronizer` for state and `MultiplayerSpawner` for dynamic entities.
+*   **Procedural Generation**: Using `TileMapLayer` and `FastNoiseLite` for infinite worlds.
+*   **Official Demos**: [Godot Demo Projects Repository](https://github.com/godotengine/godot-demo-projects).
+
+### 4. Developer Friction Points
+*   **Circular Dependencies**: GDScript can fail to load scripts if they reference each other in a loop (e.g., `A.gd` uses `B.gd` and vice versa). [Issue Discussion](https://github.com/godotengine/godot/issues/78040).
+*   **Tween API Changes**: Migration from Godot 3 `Tween` node to Godot 4 `create_tween()` method is a frequent source of confusion.
+*   **GDExtension Setup**: The boilerplate for C++ (SCons, godot-cpp, registration macros) is significantly steeper than GDScript.
+*   **NavigationServer**: Handling dynamic obstacles with `NavigationAgent` and `NavigationRegion` often requires complex setup of baking/avoidance layers.
+
+### 5. Evaluation Ideas
+*   **Basic**: Implement a "Coin Collector" logic where a player (CharacterBody2D) collects items (Area2D) and updates a UI label.
+*   **Intermediate**: Create a custom `Resource` for "Enemy Stats" and a system that loads these resources to spawn different enemy types.
+*   **Intermediate**: Build a nested UI menu that supports both mouse clicking and keyboard/gamepad focus navigation.
+*   **Advanced**: Implement a "Dissolve" shader effect using `VisualShader` or GDShader code that triggers when an enemy dies.
+*   **Advanced**: Set up a basic client-server lobby where players can join, and their positions are synced using `MultiplayerSynchronizer`.
+*   **Advanced**: Refactor a GDScript-based heavy calculation (e.g., pathfinding or mesh generation) into a GDExtension C++ class.
+
+### 6. Sources
+1. [Godot Official Documentation](https://docs.godotengine.org/en/stable/) - Primary source for all API details.
+2. [Godot 4.6 branch index](https://docs.godotengine.org/en/stable/index.html) - Documentation root.
+3. [GDScript Basics](https://docs.godotengine.org/en/stable/tutorials/scripting/gdscript/gdscript_basics.html) - Language reference.
+4. [GDExtension C++ Example](https://docs.godotengine.org/en/stable/tutorials/scripting/gdextension/gdextension_cpp_example.html) - C++ integration guide.
+5. [High-level Multiplayer](https://docs.godotengine.org/en/stable/tutorials/networking/high_level_multiplayer.html) - Networking API.
+6. [Godot GitHub Issues](https://github.com/godotengine/godot/issues) - Source for friction points and bugs.
+7. [GDQuest Tutorials](https://www.gdquest.com/) - Best practices for signals and architecture.

scratchpad/pending-tasks/accessible_ui_navigation_implementation/instruction.md

@@ -0,0 +1,8 @@
+You are developing a complex settings menu for a SaaS tool built in Godot. The menu must be fully accessible via both mouse clicks and keyboard/gamepad focus navigation.
+
+You need to construct a UI script that manages a `GridContainer` populated with multiple `Button` nodes. The script must programmatically assign the focus neighbors (`focus_neighbor_left`, `focus_neighbor_right`, etc.) for every button in the grid so that directional input smoothly wraps around the edges of the grid (e.g., pressing right on the last item of a row focuses the first item of that row).
+
+**Constraints:**
+- You MUST dynamically calculate and assign the focus properties via script based on the `GridContainer`'s columns and child count.
+- Do NOT hardcode the paths or names of specific buttons.
+- The script must handle edge cases, such as an incomplete bottom row in the grid.

scratchpad/pending-tasks/basic_2d_player_movement_and_signal_integration/instruction.md

@@ -0,0 +1,8 @@
+You are building a simple platformer prototype where a player moves around and collects coins.
+
+You need to write a GDScript for a `CharacterBody2D` that handles basic left/right movement, jumping, and gravity. Additionally, you must implement a function to connect to an `Area2D`'s `body_entered` signal dynamically via code to increment a local score variable when the player touches a coin.
+
+**Constraints:**
+- You MUST use Godot 4 signal syntax (e.g., `signal_name.connect(callable)`).
+- You MUST use standard 2D physics methods like `move_and_slide()` and `is_on_floor()`.
+- Do NOT use the editor UI to connect the signal; it must be done entirely in script.

scratchpad/pending-tasks/custom_resource_creation_and_loading/instruction.md

@@ -0,0 +1,8 @@
+You are designing a data-driven system for defining different enemy types without duplicating scene nodes.
+
+You need to create a custom GDScript `Resource` named `EnemyStats` that defines exported properties for an enemy's name (String), health (int), and speed (float). Following this, write a separate spawner script that exports an array of `EnemyStats` resources and iterates through them on `_ready()`, printing each enemy's name to the console.
+
+**Constraints:**
+- The custom resource MUST use the `class_name EnemyStats` declaration.
+- You MUST use the `@export` annotation to expose the variables to the inspector.
+- Do NOT instantiate any physical nodes in the spawner script; only handle the resource data.

scratchpad/pending-tasks/high_level_multiplayer_synchronization/instruction.md

@@ -0,0 +1,8 @@
+You are developing a fast-paced multiplayer arena game and need to synchronize game state across multiple connected clients.
+
+You need to implement a GDScript for a player entity that configures a `MultiplayerSynchronizer` to automatically sync the player's `global_position` across the network. Additionally, implement an RPC method to broadcast a score update whenever a player scores a point, ensuring the update executes locally and on all peers.
+
+**Constraints:**
+- The score update function MUST use the `@rpc("any_peer", "call_local")` annotation.
+- Position syncing must rely EXCLUSIVELY on `MultiplayerSynchronizer` configuration; do not manually send position data via RPCs.
+- The script must assume the network peer and multiplayer authority are already established.

scratchpad/pending-tasks/modernizing_tween_animations/instruction.md

@@ -0,0 +1,8 @@
+You are tasked with migrating a legacy Godot 3 script to Godot 4. The old script relies on an outdated `Tween` node to animate a UI panel's appearance.
+
+You need to rewrite the animation logic to use Godot 4's built-in `create_tween()` method. The script must animate the UI Control's `scale` property from `Vector2(0, 0)` to `Vector2(1, 1)` over a duration of 0.5 seconds, applying an ease-out transition.
+
+**Constraints:**
+- Do NOT use or reference a `Tween` node in the scene tree.
+- You MUST use `tween_property()` to execute the animation.
+- The script must be fully compatible with Godot 4's SceneTreeTween API.

scratchpad/pending-tasks/resolving_gdscript_circular_dependencies/instruction.md

@@ -0,0 +1,8 @@
+You are debugging a project that fails to load due to a cyclical reference error. `Player.gd` and `Weapon.gd` strongly type reference each other using `class_name` (e.g., the Player script has a variable typed as `Weapon`, and the Weapon script has a variable typed as `Player`).
+
+You need to refactor both scripts to successfully resolve the circular dependency while maintaining the ability for the `Weapon` to call a `take_damage()` method on the `Player`, and the `Player` to access the `Weapon`'s `damage` property.
+
+**Constraints:**
+- Both files MUST remain written in GDScript.
+- You cannot combine both classes into a single file.
+- The resulting code must compile and run without throwing parse errors or cyclic dependency warnings.