Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

SHELL := /bin/bash

.PHONY: help docker-shell docker-check-agents docker-smoke docker-run docker-parallel-run docker-setup-flydsl \
.PHONY: help docker-shell docker-check-agents docker-smoke docker-run docker-parallel-run docker-setup-flydsl docker-setup-geak \
check-docker-runner check-evaluator check-held-out check-visualization \
visualization-build visualization-serve visualization-run \
sync-perf-helpers check-perf-helpers materialize-perf-workspace \
Expand All @@ -17,7 +17,7 @@ help:
@echo "======================================================"
@echo "Docker-first workflow (the only supported path):"
@echo "make docker-shell - Enter the runtime image with repo and agent auth mounted"
@echo "make docker-check-agents - Verify the first-class host CLI selected by CONFIG"
@echo "make docker-check-agents - Verify the agent stack selected by CONFIG"
@echo " Use CONFIG=... for another config; AGENTS=... overrides it"
@echo " AGENTS=all explicitly checks all three first-class CLIs"
@echo "make docker-smoke - Verify Docker Python, ROCm tools, imports, and GPU access"
Expand All @@ -27,6 +27,7 @@ help:
@echo " On other GPUs, pass a matching CONFIG explicitly"
@echo " Images: gfx942->mi30x, gfx950->mi35x; override with AKA_DOCKER_IMAGE=..."
@echo "make docker-setup-flydsl - Install FlyDSL when absent (for flydsl2flydsl, torch2flydsl, and triton2flydsl)"
@echo "make docker-setup-geak - Install the GEAK v4 Claude Agent SDK dependency"
@echo "make check-docker-runner - Check Docker runner syntax and runtime-specific arguments"
@echo "make check-evaluator - Run centralized evaluator unit tests"
@echo "make check-held-out - Run held-out module unit tests"
Expand Down Expand Up @@ -55,7 +56,11 @@ VISUALIZATION_PORT ?= 8080
MATERIALIZE_FORCE_ARG := $(if $(filter 1 true yes,$(FORCE)),--force,)

docker-shell:
@$(DOCKER_RUNNER) shell
@if [[ -n "$(AGENTS)" ]]; then \
AKA_AGENTS="$(AGENTS)" $(DOCKER_RUNNER) shell; \
else \
$(DOCKER_RUNNER) shell; \
fi

docker-check-agents:
@AKA_AGENTS="$(AGENTS)" $(DOCKER_RUNNER) check-agents --config_name $(CONFIG)
Expand All @@ -69,11 +74,16 @@ docker-run:
docker-parallel-run:
@GPU_IDS="$(GPU_IDS)" $(DOCKER_RUNNER) parallel-run --config_name $(CONFIG) $(RUN_ARGS)

# Install FlyDSL into the container's persistent pip user-base when the selected
# Install FlyDSL into the container's persistent Python dependency target when the selected
# image does not ship it. Needed by all three FlyDSL task types.
docker-setup-flydsl:
@$(DOCKER_RUNNER) setup-flydsl

# Install the Claude Agent SDK into the persistent container Python dependency target.
# The GEAK Workflow checkout itself is bind-mounted read-only from AKA_GEAK_ROOT.
docker-setup-geak:
@$(DOCKER_RUNNER) setup-geak

check-docker-runner:
@bash tests/test_docker_benchmark.sh

Expand Down
39 changes: 28 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ AgentKernelArena/
│ ├── cursor/ # Cursor Agent CLI
│ ├── claude_code/ # Claude Code CLI
│ ├── codex/ # Codex CLI
│ ├── geak_v4/ # GEAK v4 deterministic Workflow
│ ├── geak_v3/ # GEAK HIP optimization
│ ├── geak_v3_triton/ # GEAK Triton optimization
│ ├── mini_swe_triton/ # mini-swe-agent Triton optimization
Expand Down Expand Up @@ -114,12 +115,13 @@ Each run selects one `agent.template`. Repeated runs can compare different agent
| `cursor` | Cursor Agent CLI integration |
| `claude_code` | Claude Code CLI integration |
| `codex` | Codex CLI integration |
| `geak_v4` | GEAK v4 deterministic kernel Workflow through Claude Code |
| `geak_v3` | GEAK optimization for HIP tasks |
| `geak_v3_triton` | GEAK optimization for Triton tasks |
| `mini_swe_triton` | mini-swe-agent-based Triton optimization |
| `task_validator` | Task quality validation; does not optimize kernels |

Agent-specific models, effort settings, iteration guidance, timeouts, and provider configuration live under `agents/<agent_name>/agent_config.yaml` or in the selected agent CLI. Specialized agents may require additional setup; inspect their directories and agent-specific README files where present.
Agent-specific models, effort settings, iteration guidance, timeouts, and provider configuration live under `agents/<agent_name>/agent_config.yaml` or in the selected agent CLI. Specialized agents may require additional setup; inspect their directories and agent-specific README files where present. GEAK v4 setup and its single-source task contract are documented in [agents/geak_v4/README.md](agents/geak_v4/README.md).

## Task Environments

Expand Down Expand Up @@ -188,12 +190,13 @@ installation. The npm path requires Node.js 22+ and npm. See the
[official Claude Code setup guide](https://code.claude.com/docs/en/installation)
for the current alternatives.

The repository provides three ready-to-use run configurations:
The repository provides four ready-to-use run configurations:

| Configuration | Purpose |
| --- | --- |
| `example_configs/quickstart_claude_mi300.yaml` | One Claude Code GELU task on MI300/MI300X (`gfx942`); use this for a first run on MI300-series hardware. |
| `example_configs/quickstart_claude_mi355x.yaml` | One Claude Code GELU task on MI355X (`gfx950`); use this for a first run on MI355X. |
| `example_configs/quickstart_geak_v4_mi300.yaml` | One GEAK v4 GELU task on MI300/MI300X (`gfx942`); requires the additional GEAK checkout and SDK setup. |
| `example_configs/benchmark_cursor_mi355x.yaml` | Curated 60-task Cursor Agent benchmark on MI355X; use this for a longer benchmark only after installing and authenticating Cursor Agent. |

Running `make docker-run` without `CONFIG` uses the MI300/MI300X Claude
Expand All @@ -206,6 +209,18 @@ FlyDSL tasks require FlyDSL in the container. The pinned image may already provi
make docker-setup-flydsl
```

GEAK v4 requires a GEAK checkout beside AgentKernelArena by default, Claude
Code 2.1.177 or newer logged in on the host, and the Agent SDK:

```bash
make docker-setup-geak
```

Set `AKA_GEAK_ROOT=/absolute/path/to/GEAK` when the checkout is not the default
sibling directory. This target installs only `claude-agent-sdk`; it does not
`pip install` GEAK, which is mounted read-only. See the
[GEAK v4 agent README](agents/geak_v4/README.md) before selecting its example.

Performance timing helpers are maintained in `src/tools/perf/` and materialized into run workspaces. See [src/tools/perf/README.md](src/tools/perf/README.md) before changing task timing code.

For detailed installation and compatibility information, see [docs/install/install.md](docs/install/install.md) and [docs/reference/compatibility-matrix.md](docs/reference/compatibility-matrix.md).
Expand All @@ -227,18 +242,19 @@ Run agent-specific settings such as `model`, `effort`, `max_iterations`, and
`timeout_seconds` are configured in the selected agent's `agent_config.yaml`,
not in the run configuration.

For a Cursor, Claude Code, Codex, or task-validator config, verify only the
selected first-class host CLI (the validator resolves to its configured backend):
For a Cursor, Claude Code, Codex, GEAK v4, or task-validator config, verify only
the dependencies selected by the config (the validator resolves to its
configured backend):

```bash
CONFIG_PATH=my_experiment.yaml
make docker-check-agents CONFIG="$CONFIG_PATH"
```

Use `AGENTS=claude_code,codex` to check an explicit subset or `AGENTS=all` to
check Cursor, Claude Code, and Codex together. Specialized integrations such as
GEAK and mini-swe have their own dependency checks and are not handled by this
command.
Use `AGENTS=claude_code,codex` to check an explicit subset,
`AGENTS=geak_v4` to check the GEAK v4 stack, or `AGENTS=all` to check Cursor,
Claude Code, and Codex together. Legacy GEAK and mini-swe integrations retain
their own dependency checks.

### Run Serially

Expand All @@ -258,9 +274,10 @@ make docker-parallel-run CONFIG="$CONFIG_PATH"
```

The Docker parallel path is verified for `cursor`, `claude_code`, `codex`, and
`task_validator`. Specialized GEAK/mini-swe integrations need their own
dependencies and GPU-ID configuration before they are used with isolated
workers.
`task_validator`. `geak_v4` maps each isolated worker to logical GPU 0, but
still requires the setup in its agent README. A paid Claude workflow invocation
is not part of the integration's offline validation. Legacy GEAK/mini-swe
integrations need their own dependencies and GPU-ID configuration.

### Resume a Run

Expand Down
127 changes: 127 additions & 0 deletions agents/geak_v4/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# GEAK v4 Agent

The `geak_v4` integration runs GEAK's deterministic
`kernel_workflow/kernel_workflow.js` through Claude Code's dynamic Workflow
tool. GEAK works on a disposable copy of the task workspace; AgentKernelArena
imports only a validated patch for the one declared kernel source and then runs
its normal correctness and performance evaluation.

## Prerequisites

- An AMD Instinct GPU and `rocprof-compute`, as required by the Arena Docker
smoke/run contract.
- A local GEAK checkout. By default, the Docker runner looks for `GEAK` beside
the Arena checkout:

```text
parent/
├── AgentKernelArena/
└── GEAK/
```

- Claude Code 2.1.177 or newer, installed and logged in on the host. The minimum
version is required for the dynamic Workflow feature.
- Access to the Claude model configured in
`agents/geak_v4/agent_config.yaml`.
- FlyDSL installed with `make docker-setup-flydsl` when selecting a
`flydsl2flydsl` task.

## Setup

Clone GEAK beside AgentKernelArena, authenticate Claude Code on the host, and
install the Python SDK into the persistent container dependency directory:

```bash
cd /path/to/parent
git clone https://github.com/AMD-AGI/GEAK.git
cd AgentKernelArena

claude --version
claude
claude auth status

make docker-setup-geak
```

This adapter was developed against GEAK commit
`4965d5b2ccde927925c8c5501a25c1233daa52eb`
(`v4.0.0-102-g4965d5b`). For a reproducible review, check out that revision;
newer GEAK revisions may require an adapter/schema update.

If the GEAK checkout is elsewhere, provide its absolute host path for setup,
preflight, and every run:

```bash
export AKA_GEAK_ROOT=/absolute/path/to/GEAK
make docker-setup-geak
```

`make docker-setup-geak` installs only `claude-agent-sdk`. It does **not**
`pip install` GEAK. The Docker runner bind-mounts the checkout read-only at
`/opt/geak`, so the workflow code, roles, and knowledge remain unchanged by an
Arena run. The persistent Python dependency directory is writable only in the
explicit setup container and is nested-mounted read-only during agent runs.

## Run the example

The included MI300/MI300X example runs one HIP GELU task:

```bash
CONFIG_PATH=example_configs/quickstart_geak_v4_mi300.yaml
make docker-check-agents CONFIG="$CONFIG_PATH"
make docker-run CONFIG="$CONFIG_PATH"
```

`docker-check-agents` verifies the Claude login and version, the Agent SDK, the
GEAK workflow checkout, and an available profiler without starting an
optimization.

## V1 supported task contract

V1 of the Arena integration supports:

- `hip2hip`
- `triton2triton`
- `flydsl2flydsl`

A task must declare exactly one existing source in `source_file_path`. That
source must be a normal, non-symlink file and cannot be a protected config,
test, or harness path. In particular, names such as `test_*.py`, `*_test.py`,
`test_*.cpp`, `*_test.hip`, and `*_harness.cu` are rejected. Tasks with
multiple sources or a source that also contains the test harness are
intentionally out of scope.

Authoring, translation, repository, and image-level tasks are not supported by
V1. Use a supported task with a single standalone kernel source.

## Isolation and artifacts

For a task workspace named `<workspace>`, GEAK artifacts are kept outside the
scored workspace under:

```text
.<workspace>_geak_v4/<run-id>/
├── input/ # Disposable task copy seen by GEAK
├── eval/ # GEAK validation and final patch artifacts
├── runs/ # Workflow experiment artifacts
├── handoff.json
└── result.json
```

The hidden artifact directory is a sibling of the task workspace. GEAK is
instructed not to apply changes to the original input, and the Arena accepts a
result only when GEAK's Director validation passes and the final patch changes
the one allowlisted source without creating, deleting, renaming, or changing
the mode of a file. Arena also compares a full manifest of the scored workspace
before and after Workflow execution; a direct mutation makes the task fail
before scoring or patch import.

## Validation boundary

Offline tests cover handoff mapping, result parsing, patch filtering, artifact
isolation, and Docker argument construction without contacting Claude. A real
paid Claude workflow invocation is not part of that offline validation; run
the one-task example with an authorized account before relying on this
integration for a benchmark campaign. As with other Arena task workspaces, the
disposable-copy and manifest checks are fail-closed integrity controls, not an
OS security sandbox; a failed run is not automatically rolled back.
4 changes: 4 additions & 0 deletions agents/geak_v4/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Copyright(C) [2026] Advanced Micro Devices, Inc. All rights reserved.
from .launch_agent import launch_agent

__all__ = ["launch_agent"]
29 changes: 29 additions & 0 deletions agents/geak_v4/agent_config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# GEAK v4 runs the deterministic kernel_workflow through Claude Code's dynamic
# Workflow tool. The GEAK checkout is mounted read-only at /opt/geak; all run
# artifacts are written beside the Arena task workspace.
workflow_dir: /opt/geak/kernel_workflow

# Defaults from GEAK v4's supported standalone kernel workflow.
model: claude-opus-4-8
effort: ultracode
budget: 6
min_improve: 0.02
deep_cost: 2
use_expert_skills: false

# Hard wall-clock bound for the complete multi-agent workflow.
timeout_seconds: 43200

# Newer Claude Code builds may finish detached validation work after the
# background task notification. Keep the SDK client alive for this bounded
# on-disk completion grace period.
done_grace_seconds: 1800
done_poll_seconds: 5

# V1 deliberately targets existing, isolated kernels with one declared source
# file. Author/translation tasks need a separate frozen-baseline adapter, while
# repository/image tasks need a storage/copy-cost qualification first.
supported_task_types:
- hip2hip
- triton2triton
- flydsl2flydsl
Loading
Loading