Initialize repository with benchmark template

.github/workflows/deploy.yml

@@ -0,0 +1,69 @@
+name: Deploy Next.js site to Pages
+
+on:
+  push:
+    branches: ["main"]
+  pull_request:
+    types: [opened, reopened, synchronize, closed]
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pull-requests: write
+  pages: write
+  id-token: write
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.ref_name }}
+  cancel-in-progress: true
+
+jobs:
+  build-and-deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Install Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: 22
+
+      - name: Setup Bun
+        uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: latest
+
+      - name: Install dependencies
+        run: bun install --frozen-lockfile
+        working-directory: site
+
+      - name: Setup Pages
+        id: setup_pages
+        uses: actions/configure-pages@v5
+
+      - name: Set base uri
+        run: |
+          if [ "${{ github.event_name }}" = "pull_request" ]; then
+            echo "NEXT_PUBLIC_BASE_PATH=${{ steps.setup_pages.outputs.base_path }}/pr-preview/pr-${{ github.event.pull_request.number }}" >> "$GITHUB_ENV"
+          else
+            echo "NEXT_PUBLIC_BASE_PATH=${{ steps.setup_pages.outputs.base_path }}" >> "$GITHUB_ENV"
+          fi
+
+      - name: Build with Next.js
+        run: bun run build
+        working-directory: site
+
+      - name: Deploy preview
+        if: github.event_name == 'pull_request'
+        uses: rossjrw/pr-preview-action@v1
+        with:
+          source-dir: site/out
+
+      - name: Deploy production
+        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+        uses: JamesIves/github-pages-deploy-action@v4
+        with:
+          clean-exclude: pr-preview/
+          force: false
+          folder: site/out

.gitignore

@@ -0,0 +1,49 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+node_modules
+.pnp
+.pnp.*
+.yarn/*
+!.yarn/patches
+!.yarn/plugins
+!.yarn/releases
+!.yarn/versions
+
+__pycache__/
+*.py[cod]
+*$py.class
+
+# testing
+coverage
+
+# next.js
+.next/
+
+# The `out` directory should not be ignored by version control
+out/
+
+# production
+build
+
+# misc
+.DS_Store
+*.pem
+*~
+\#*
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# local env files
+.env*.local
+.env*
+
+# vercel
+.vercel
+
+# typescript
+*.tsbuildinfo
+next-env.d.ts

.zealt/config.json

@@ -0,0 +1,5 @@
+{
+  "title": "Altair Benchmark",
+  "description": "Performance results of AI coding models on Altair tasks, measuring success rate and execution time with high precision.",
+  "github_repo": "https://github.com/kweizh/altair-benchmark"
+}

README.md

@@ -1 +1,46 @@
-# altair-benchmark
+
+# Altair Benchmark
+
+This repository contains benchmarks for evaluating AI models on **Altair**.
+
+You can view the evaluation reports at [https://kweizh.github.io/altair-benchmark/](https://kweizh.github.io/altair-benchmark/).
+
+## Project Structure
+
+- `tasks/`: Contains the benchmark tasks, each with its own instructions.
+- `jobs/`: Stores the results of benchmark runs.
+- `site/`: A Next.js application to visualize benchmark results.
+
+## Getting Started
+
+This benchmark is evaluated using the [Harbor framework](https://github.com/harbor-framework/harbor) and the [Pochi agent](https://github.com/TabbyML/pochi).
+
+### Running Evaluation
+
+You can run the evaluation using the Harbor CLI. Here is an example:
+
+```bash
+harbor run \
+  --agent codex \
+  --model "gpt-5.2-codex" \
+  --env daytona \
+  --path ./tasks \
+  --n-attempts 1 \
+  --max-retries 5 \
+  --n-concurrent 5 \
+  --retry-include RuntimeError \
+  --retry-include DaytonaError \
+  --retry-include AgentTimeoutError
+```
+
+### Evaluation Details
+
+Before starting the evaluation, you should set the necessary environment variables for your chosen agent.
+For example, if using Pochi, you should export `POCHI_API_KEY`.
+
+Evaluation can be run locally with Docker (default), or using [Daytona.io](https://www.daytona.io/) by setting `--env daytona`.
+
+When running with Daytona, please note that Daytona blocks some network access for tier 1 and tier 2 users. If you meet any network issues, please refer to [Daytona network limits](https://www.daytona.io/docs/en/network-limits/).
+
+---
+Generated by [Zealt](https://github.com/TabbyML/zealt)

plan.md

@@ -0,0 +1,103 @@
+# Evaluation Dataset Research: Altair (Vega-Altair)
+
+## 1. Library Overview
+*   **Description**: Altair is a declarative statistical visualization library for Python, built on top of the [Vega-Lite](https://vega.github.io/vega-lite/) grammar. It allows users to describe visualizations in terms of data transformations and visual encodings rather than low-level imperative drawing commands.
+*   **Ecosystem Role**: It is the standard declarative plotting library for the Python data science stack (Pandas, Polars, NumPy). It integrates deeply with Jupyter, VS Code, and Streamlit.
+*   **Project Setup**:
+    ```bash
+    pip install altair vega_datasets
+    # For large datasets (optional but recommended)
+    pip install vegafusion[all] 
+    ```
+
+## 2. Core Primitives & APIs
+
+### Key Objects
+*   **`alt.Chart(data)`**: The fundamental object. Data can be a Pandas DataFrame, Polars DataFrame, or a URL string pointing to a JSON/CSV file.
+*   **`mark_*()`**: Defines the geometry (e.g., `mark_point()`, `mark_bar()`, `mark_line()`, `mark_area()`, `mark_rect()`, `mark_text()`).
+*   **`encode()`**: Maps data columns to visual channels (e.g., `x`, `y`, `color`, `size`, `shape`, `tooltip`).
+*   **`add_params()`**: (v5+) Replaces `add_selection`. Used to add interactive parameters like selections to a chart.
+
+### Code Examples
+
+#### Basic Chart with Shorthand Types
+Altair uses a shorthand syntax for data types: `:Q` (Quantitative), `:N` (Nominal), `:O` (Ordinal), `:T` (Temporal).
+```python
+import altair as alt
+from vega_datasets import data
+
+cars = data.cars()
+chart = alt.Chart(cars).mark_point().encode(
+    x='Horsepower:Q',
+    y='Miles_per_Gallon:Q',
+    color='Origin:N',
+    tooltip=['Name', 'Origin']
+).interactive() # Enables zoom/pan
+```
+
+#### Composition (Layering & Concatenation)
+*   **Layering (`+`)**: Overlays charts.
+*   **Horizontal Concatenation (`|`)**: Side-by-side.
+*   **Vertical Concatenation (`&`)**: Top-to-bottom.
+```python
+base = alt.Chart(cars).encode(x='Horsepower:Q')
+layers = base.mark_bar() + base.mark_rule(color='red').transform_aggregate(x='mean(Horsepower)')
+concat = (chart1 | chart2).properties(title="Side by Side")
+```
+
+#### Advanced Interaction (v5+ Syntax)
+Using `selection_interval` for cross-filtering.
+```python
+brush = alt.selection_interval()
+
+points = alt.Chart(cars).mark_point().encode(
+    x='Horsepower:Q',
+    y='Miles_per_Gallon:Q',
+    color=alt.when(brush).then('Origin:N').otherwise(alt.value('lightgray'))
+).add_params(brush)
+
+bars = alt.Chart(cars).mark_bar().encode(
+    y='Origin:N',
+    x='count()',
+    color='Origin:N'
+).transform_filter(brush)
+
+dashboard = points & bars
+```
+
+#### Transformations
+```python
+chart.transform_calculate(
+    Efficiency='datum.Miles_per_Gallon / datum.Weight'
+).transform_filter(
+    alt.datum.Efficiency > 0.01
+)
+```
+
+## 3. Real-World Use Cases & Templates
+*   **Interactive Dashboards**: Linked views where selecting data in one plot (e.g., a map or timeline) filters the others.
+*   **Statistical Exploration**: Visualizing distributions with binned histograms and box plots.
+*   **Geographic Mapping**: Using `mark_geoshape` with TopoJSON data.
+*   **Integration with Streamlit**: Using `st.altair_chart` to build data apps with bidirectional communication (selections in Altair updating Streamlit state).
+
+## 4. Developer Friction Points
+1.  **MaxRowsError**: By default, Altair limits datasets to 5,000 rows to prevent browser crashes.
+    *   *Solution*: Use `alt.data_transformers.disable_max_rows()` or `alt.data_transformers.enable('vegafusion')`.
+2.  **Date Handling**: Passing Python `datetime` objects in selections or filters can sometimes fail if not converted correctly by the underlying Vega-Lite engine.
+3.  **Complex Faceting**: Faceted charts with `resolve_scale(y='independent')` can be tricky when combined with shared selections across facets.
+4.  **Encoding Conflicts**: Forgetting to specify data types (e.g., `:O` vs `:N`) can lead to unexpected axis sorting or color scales.
+
+## 5. Evaluation Ideas
+*   **Simple**: Create a bar chart of average MPG by Origin with sorted bars and custom tooltips.
+*   **Intermediate**: Build a layered plot showing a scatter plot of data points with a regression line (using `transform_regression`).
+*   **Intermediate**: Implement an interactive "Focus + Context" chart (a small overview chart with a brush that controls the X-axis of a larger detail chart).
+*   **Complex**: Create a cross-filtering dashboard with three linked views (Scatter, Histogram, and Heatmap) using `selection_point` and `selection_interval`.
+*   **Complex**: Design a geographic map of US airports where clicking an airport highlights its connections on the same map (using `transform_lookup`).
+*   **Edge Case**: Handle a 50,000-row dataset by configuring `VegaFusion` and implementing an aggregated heatmap to avoid browser lag.
+
+## 6. Sources
+1.  [Official Vega-Altair Documentation](https://altair-viz.github.io/): Main reference for API and User Guide.
+2.  [Altair GitHub Repository](https://github.com/vega/altair): Source for issues and release notes.
+3.  [Vega-Lite Documentation](https://vega.github.io/vega-lite/): Documentation for the underlying grammar.
+4.  [VegaFusion Documentation](https://vegafusion.io/): Solutions for large dataset scaling.
+5.  [Streamlit Altair Integration](https://docs.streamlit.io/develop/api-reference/charts/st.altair_chart): Guide for interactive web apps.

scratchpad/pending-tasks/basic_aggregation_and_sorting_chart/instruction.md

@@ -0,0 +1,9 @@
+Visualizing aggregate metrics and standardizing encodings is a fundamental first step in exploratory data analysis with Altair. Altair's shorthand syntax simplifies defining data types, but explicit sorting and tooltips require careful configuration.
+
+You need to create a bar chart showing the average "Miles_per_Gallon" by "Origin" using the `cars` dataset from `vega_datasets` in a standard Python environment. 
+
+**Constraints:**
+- Must explicitly use Altair shorthand types (e.g., `:Q` for Quantitative, `:N` for Nominal).
+- Bars MUST be sorted in descending order based on the average MPG.
+- Tooltips must be added to display both the "Origin" and the computed average MPG.
+- Save the resulting chart specification to a file named `bar_chart.json`.

scratchpad/pending-tasks/focus_and_context_interactive_chart/instruction.md

@@ -0,0 +1,9 @@
+Interactive "Focus + Context" charts allow users to zoom in on specific temporal or quantitative data regions without losing sight of the overall trend, leveraging Altair's selection APIs.
+
+You need to implement an interactive chart using Altair v5+ syntax where a small overview area chart includes an interval brush that dynamically controls the X-axis domain of a larger detailed line chart.
+
+**Constraints:**
+- MUST use Altair v5+ syntax, specifically `alt.selection_interval()` and `add_params()` (do NOT use the deprecated `add_selection`).
+- The two charts must be vertically concatenated using the `&` operator.
+- The X-axis of the detail chart must strictly bind to the brush parameter from the overview chart.
+- Save the resulting chart specification to `focus_context.json`.

scratchpad/pending-tasks/independent_facet_scales_with_shared_selection/instruction.md

@@ -0,0 +1,9 @@
+Combining faceted charts with shared interactive selections often leads to axis scale conflicts or unexpected behaviors if scales are not properly resolved.
+
+You need to create a faceted scatter plot (faceted by "Origin" into separate columns) of the `cars` dataset where a legend-bound point selection highlights specific "Cylinders" across all facets simultaneously.
+
+**Constraints:**
+- You MUST ensure the Y-axis is optimized for each facet by using `resolve_scale(y='independent')`.
+- You MUST use `alt.selection_point(fields=['Cylinders'], bind='legend')` to create the interactivity.
+- The opacity of points not matching the legend selection must drop to `0.2` across all facets.
+- Save the resulting chart specification to `faceted_shared.json`.

scratchpad/pending-tasks/large_dataset_handling_with_vegafusion/instruction.md

@@ -0,0 +1,9 @@
+Altair restricts datasets over 5,000 rows by default (throwing a `MaxRowsError`) to prevent browser crashes from massive JSON payloads. Addressing this is critical for production-grade data science pipelines.
+
+You need to process a synthesized Pandas DataFrame containing 50,000 rows and visualize it as an aggregated 2D heatmap showing record counts.
+
+**Constraints:**
+- You MUST explicitly bypass the 5,000-row limit by configuring the environment with `alt.data_transformers.enable('vegafusion')`.
+- Do NOT use `alt.data_transformers.disable_max_rows()` as it risks browser lock-up.
+- The visualization must aggregate the data into bins on both the X and Y axes within Altair (do not pre-bin in Pandas).
+- Save the resulting visualization as `heatmap_large.html`.

scratchpad/pending-tasks/layered_scatter_plot_with_regression_line/instruction.md

@@ -0,0 +1,9 @@
+Altair excels at visual composition, allowing developers to overlay analytical transformations directly on top of raw data visualizations using the layering operator (`+`).
+
+You need to build a layered chart containing a base scatter plot of "Horsepower" (X-axis) versus "Miles_per_Gallon" (Y-axis) from the `cars` dataset, and overlay a linear regression line in a Python script.
+
+**Constraints:**
+- The regression line MUST be calculated natively within Altair using `transform_regression`.
+- The scatter plot points and the regression line must be distinct colors (e.g., blue for points, red for the line).
+- Do NOT pre-calculate the regression line using Pandas or NumPy.
+- Save the rendered chart to a file named `regression_chart.html`.

scratchpad/pending-tasks/tri_view_cross_filtering_dashboard/instruction.md

@@ -0,0 +1,9 @@
+Dashboards often require linked views where selecting or brushing data in one visual updates the subset of data displayed in the others, providing deep multi-dimensional exploration.
+
+You need to create a cross-filtering dashboard comprising three linked views (a scatter plot, a bar chart, and a histogram) using the `cars` dataset. 
+
+**Constraints:**
+- Apply an interval selection brush (`alt.selection_interval()`) to the scatter plot.
+- The bar chart and histogram MUST dynamically filter their displayed data based on the scatter plot's brush using `transform_filter`.
+- Unselected points in the scatter plot should turn `lightgray` using an `alt.when().then().otherwise()` condition.
+- Save the complete dashboard layout to `dashboard.html`.