From f9c059c79ddda2aee5e0bf4944757157d77e8912 Mon Sep 17 00:00:00 2001 From: yueliu14 Date: Mon, 27 Jul 2026 09:45:22 +0000 Subject: [PATCH 1/4] Add lean GEAK v4 agent integration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Register a `geak_v4` agent that drives GEAK's kernel_workflow via an SDK-based background-Workflow runner and applies the validated patch in place (apply_to_original="true"), letting the Arena harness enforce integrity and re-score. All changes are confined to agents/geak_v4/, example_configs/, module_registration, and tests — the runner is configured purely through environment variables (GEAK_V4_WORKFLOW_DIR) with no changes to Docker/Makefile/docs, mirroring the `forge` agent. Co-Authored-By: Claude Opus 4.8 --- agents/geak_v4/README.md | 108 +++ agents/geak_v4/__init__.py | 4 + agents/geak_v4/agent_config.yaml | 30 + agents/geak_v4/launch_agent.py | 377 ++++++++ agents/geak_v4/workflow_runner.py | 876 ++++++++++++++++++ example_configs/quickstart_geak_v4_mi300.yaml | 11 + src/module_registration.py | 9 +- tests/test_geak_v4.py | 496 ++++++++++ 8 files changed, 1908 insertions(+), 3 deletions(-) create mode 100644 agents/geak_v4/README.md create mode 100644 agents/geak_v4/__init__.py create mode 100644 agents/geak_v4/agent_config.yaml create mode 100644 agents/geak_v4/launch_agent.py create mode 100644 agents/geak_v4/workflow_runner.py create mode 100644 example_configs/quickstart_geak_v4_mi300.yaml create mode 100644 tests/test_geak_v4.py diff --git a/agents/geak_v4/README.md b/agents/geak_v4/README.md new file mode 100644 index 00000000..fdf2b86a --- /dev/null +++ b/agents/geak_v4/README.md @@ -0,0 +1,108 @@ +# GEAK v4 Agent + +The `geak_v4` integration runs GEAK's deterministic +`kernel_workflow/kernel_workflow.js` through Claude Code's dynamic **Workflow** +tool. GEAK optimizes the kernel and, when its Director validation passes, applies +the validated patch directly into the task workspace (`apply_to_original="true"`). + +**AgentKernelArena is the single source of truth for scoring.** After GEAK +finishes, AKA verifies its harness is intact, re-materializes the perf helpers, +and independently re-evaluates the kernel (compile → correctness → performance); +GEAK's own numbers are not used for the final result. + +Like the `forge` and `claude_code` agents, `geak_v4` relies purely on the +environment — it touches nothing outside `agents/geak_v4/`. The launcher stays +thin: it writes a versioned handoff and shells out to `workflow_runner.py`. + +## Why a runner instead of a plain `claude -p` + +On current Claude builds the `Workflow` tool runs as a **background task**: the +main agent turn ends immediately, so a one-shot `claude -p` would return (and +tear the still-running workflow down) before GEAK finishes. `workflow_runner.py` +keeps a persistent `claude_agent_sdk` client alive and drives completion off the +SDK's background-task lifecycle (`TaskStartedMessage` → `TaskNotificationMessage`) +plus GEAK's on-disk terminal marker. It is a kernel-scoped analogue of GEAK's own +`interface/run_e2e.py` and mirrors its `handoff.json` / `result.json` contract. + +## Prerequisites + +- An AMD Instinct GPU and a supported profiler (`rocprof-compute`). +- A local GEAK checkout. Point `GEAK_V4_WORKFLOW_DIR` at its + `kernel_workflow/` directory (default: `/opt/geak/kernel_workflow`). +- Claude Code 2.1.177 or newer, installed and logged in (the minimum version for + the dynamic Workflow feature), plus access to the model configured in + `agents/geak_v4/agent_config.yaml`. +- The `claude-agent-sdk` Python package installed in the interpreter that runs + the agent. +- For a `flydsl2flydsl` task, FlyDSL available in the environment. + +## Setup + +```bash +# 1. Clone GEAK and expose its kernel_workflow directory. +git clone https://github.com/AMD-AGI/GEAK.git +export GEAK_V4_WORKFLOW_DIR=/absolute/path/to/GEAK/kernel_workflow + +# 2. Authenticate Claude Code and confirm the Workflow-capable version. +claude --version +claude # log in +claude auth status + +# 3. Install the SDK the runner uses to hold the background Workflow open. +pip install claude-agent-sdk +``` + +This adapter was developed against GEAK commit +`4965d5b2ccde927925c8c5501a25c1233daa52eb` (`v4.0.0-102-g4965d5b`). For a +reproducible review, check out that revision; newer GEAK revisions may require an +adapter/schema update. + +## Run the example + +The included MI300/MI300X example runs one HIP GELU task. Run it in an +environment that already satisfies the prerequisites above (`GEAK_V4_WORKFLOW_DIR` +set, `claude-agent-sdk` installed, `claude` logged in): + +```bash +python main.py --config_name example_configs/quickstart_geak_v4_mi300.yaml +``` + +## Supported task types + +- `hip2hip` +- `triton2triton` +- `flydsl2flydsl` + +These are single standalone kernels. The launcher reads `source_file_path` to +steer the optimizer ("optimize only these files") and fails early if the declared +anchor source is missing. Authoring, translation, repository, and image-level +tasks are out of scope for this integration. + +## Artifacts + +For a task workspace named ``, GEAK's run artifacts live OUTSIDE the +scored workspace, under a hidden sibling directory: + +```text +._geak_v4// +├── eval/ # GEAK validation + final_patch.diff +├── runs/ # GEAK experiment tree (its own copy of the kernel) +├── handoff.json +└── result.json +``` + +They sit beside the workspace (not inside it) because GEAK copies `kernel_path` +into its own experiment tree — nesting outputs under the workspace would recurse, +and it keeps GEAK's scratch clear of the directory Arena scores. + +## Integrity boundary + +`geak_v4` lets GEAK edit the workspace in place, exactly like `forge` and +`claude_code`. Integrity is enforced by the Arena harness, not the launcher: +`main.py` snapshots the harness before the run, verifies it afterwards, +re-materializes the perf helpers, and re-scores the kernel independently. A run +that tampers with a protected harness path fails harness verification. This is a +fail-closed correctness control, not an OS security sandbox; a failed run is not +automatically rolled back. A real paid Claude workflow invocation is not part of +the offline tests — run the one-task example with an authorized account before +relying on this integration for a benchmark campaign. diff --git a/agents/geak_v4/__init__.py b/agents/geak_v4/__init__.py new file mode 100644 index 00000000..c54dbfb2 --- /dev/null +++ b/agents/geak_v4/__init__.py @@ -0,0 +1,4 @@ +# Copyright(C) [2026] Advanced Micro Devices, Inc. All rights reserved. +from .launch_agent import launch_agent + +__all__ = ["launch_agent"] diff --git a/agents/geak_v4/agent_config.yaml b/agents/geak_v4/agent_config.yaml new file mode 100644 index 00000000..4dbd5770 --- /dev/null +++ b/agents/geak_v4/agent_config.yaml @@ -0,0 +1,30 @@ +# GEAK v4 runs the deterministic kernel_workflow through Claude Code's dynamic +# Workflow tool. Point the runner at your GEAK checkout's kernel_workflow/ +# directory; the GEAK_V4_WORKFLOW_DIR environment variable overrides this value. +# All run artifacts are written to a hidden sibling of the Arena task workspace. +workflow_dir: /opt/geak/kernel_workflow + +# Defaults from GEAK v4's supported standalone kernel workflow. +model: claude-opus-4-8 +effort: ultracode +budget: 6 +min_improve: 0.02 +deep_cost: 2 +use_expert_skills: false + +# Hard wall-clock bound for the complete multi-agent workflow. +timeout_seconds: 43200 + +# Newer Claude Code builds may finish detached validation work after the +# background task notification. Keep the SDK client alive for this bounded +# on-disk completion grace period. +done_grace_seconds: 1800 +done_poll_seconds: 5 + +# V1 deliberately targets existing, isolated kernels with one declared source +# file. Author/translation tasks need a separate frozen-baseline adapter, while +# repository/image tasks need a storage/copy-cost qualification first. +supported_task_types: + - hip2hip + - triton2triton + - flydsl2flydsl diff --git a/agents/geak_v4/launch_agent.py b/agents/geak_v4/launch_agent.py new file mode 100644 index 00000000..a6ffdb27 --- /dev/null +++ b/agents/geak_v4/launch_agent.py @@ -0,0 +1,377 @@ +# Copyright(C) [2026] Advanced Micro Devices, Inc. All rights reserved. +"""AgentKernelArena adapter for GEAK v4's deterministic kernel workflow. + +GEAK v4 optimizes a kernel by running its JavaScript ``kernel_workflow`` through +Claude Code's dynamic ``Workflow`` tool. On current Claude builds that Workflow +runs as a *background* task, so the SDK / background-task lifecycle is handled by +the sibling ``workflow_runner.py`` (a kernel-scoped analogue of GEAK's own +``interface/run_e2e.py``). This launcher stays deliberately thin and, like the +``forge`` / ``claude_code`` agents, relies purely on the environment: + + * gate unsupported task types, + * resolve the GEAK checkout (``GEAK_V4_WORKFLOW_DIR``) and ``claude`` (PATH), + * write a versioned handoff and run ``workflow_runner.py`` as a subprocess, + * let GEAK apply its Director-validated patch straight into the workspace + (``apply_to_original="true"``). + +Workspace integrity is the Arena harness's job, not the launcher's: main.py +snapshots the harness before the agent runs, verifies it afterwards, +re-materializes the perf helpers, and independently re-scores the kernel. That is +why this adapter needs no disposable-copy / manifest / patch-reimport machinery. +""" + +from __future__ import annotations + +import json +import logging +import os +import shutil +import signal +import subprocess +import sys +import threading +import time +import uuid +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +import yaml + +from agents import register_agent +from src.module_registration import AgentType, load_prompt_builder + + +_JSON_SIZE_LIMIT = 8 * 1024 * 1024 +_PROCESS_OUTPUT_LIMIT = 4 * 1024 * 1024 + + +def _load_yaml(path: str | Path) -> dict[str, Any]: + with Path(path).open("r", encoding="utf-8") as stream: + return yaml.safe_load(stream) or {} + + +def _read_json(path: Path) -> dict[str, Any] | None: + try: + content = path.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError): + return None + if len(content) > _JSON_SIZE_LIMIT: + return None + try: + value = json.loads(content) + except json.JSONDecodeError: + return None + return value if isinstance(value, dict) else None + + +def _declared_sources(task_config: dict[str, Any], workspace: Path) -> list[str]: + """Return the workspace-relative source files used to steer the optimizer. + + Only guidance: GEAK edits the kernel and the Arena harness re-scores it, so + this is not a security boundary. We fail early if the declared anchor source + is absent (mirrors ``forge``'s kernel-file resolution) and otherwise pass the + names through to the "optimize only these files" prompt note. + """ + raw = task_config.get("source_file_path") or [] + values = [raw] if isinstance(raw, str) else list(raw) + sources = [value.strip() for value in values if isinstance(value, str) and value.strip()] + if sources and not (workspace / sources[0]).exists(): + raise FileNotFoundError( + f"declared source_file_path not found in workspace: {sources[0]}" + ) + return sources + + +def _logical_gpu_ids(eval_config: dict[str, Any]) -> str: + """Return GPU IDs in the process-visible namespace. + + Arena parallel workers mask one physical GPU with ROCR_VISIBLE_DEVICES and + expose it as logical HIP/CUDA device 0. GEAK's gpu_lock wrapper rewrites + HIP_VISIBLE_DEVICES again, so forwarding the host ID would hide the GPU. + """ + if os.environ.get("AGENT_KERNEL_ARENA_HOST_GPU_ID") is not None: + return "0" + + visible = ( + os.environ.get("HIP_VISIBLE_DEVICES") + or os.environ.get("CUDA_VISIBLE_DEVICES") + ) + if visible: + count = len([part for part in visible.split(",") if part.strip()]) + if count: + return ",".join(str(index) for index in range(count)) + + override = os.environ.get("GEAK_V4_GPU_IDS") + configured = override if override is not None else eval_config.get("gpu_ids", "0") + if isinstance(configured, (list, tuple)): + configured = ",".join(str(item) for item in configured) + return str(configured) + + +def _process_group_exists(pgid: int) -> bool: + try: + os.killpg(pgid, 0) + except ProcessLookupError: + return False + except PermissionError: + return True + return True + + +def _wait_group_exit(process: subprocess.Popen[str], pgid: int, timeout: float) -> bool: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + process.poll() + if not _process_group_exists(pgid): + return True + time.sleep(0.1) + return not _process_group_exists(pgid) + + +def _terminate_process_group( + process: subprocess.Popen[str], + logger: logging.Logger, +) -> None: + pgid = process.pid + if not _process_group_exists(pgid): + process.poll() + return + try: + os.killpg(pgid, signal.SIGTERM) + except ProcessLookupError: + return + if _wait_group_exit(process, pgid, 10): + return + logger.warning("Force killing GEAK runner process group") + try: + os.killpg(pgid, signal.SIGKILL) + except ProcessLookupError: + return + _wait_group_exit(process, pgid, 5) + + +def _stream_pipe(stream, prefix: str, output: list[str], log) -> None: + captured = 0 + truncated = False + try: + while True: + chunk = stream.read(4096) + if not chunk: + break + remaining = _PROCESS_OUTPUT_LIMIT - captured + if remaining > 0: + retained = chunk[:remaining] + output.append(retained) + captured += len(retained) + compact = " ".join(retained[:2000].split()) + if compact: + log(f"{prefix} {compact[:500]}") + if len(chunk) > remaining and not truncated: + truncated = True + log(f"{prefix} output truncated at {_PROCESS_OUTPUT_LIMIT} characters") + finally: + stream.close() + + +def _run_workflow_runner( + handoff_path: Path, + result_path: Path, + *, + timeout_seconds: int, + logger: logging.Logger, +) -> str: + """Run workflow_runner.py in its own session and stream its output. + + The runner keeps the Claude SDK client alive until GEAK's background Workflow + completes; ``start_new_session=True`` lets us tear down the whole group + (runner + claude + any background Workflow child) on timeout or error. + """ + runner = Path(__file__).with_name("workflow_runner.py") + command = [sys.executable, str(runner), str(handoff_path), str(result_path)] + process = subprocess.Popen( + command, + cwd=str(handoff_path.parent), + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + errors="replace", + bufsize=1, + start_new_session=True, + ) + assert process.stdout is not None + assert process.stderr is not None + stdout: list[str] = [] + stderr: list[str] = [] + stdout_thread = threading.Thread( + target=_stream_pipe, + args=(process.stdout, "[GEAK]", stdout, logger.info), + daemon=True, + ) + stderr_thread = threading.Thread( + target=_stream_pipe, + args=(process.stderr, "[GEAK STDERR]", stderr, logger.warning), + daemon=True, + ) + stdout_thread.start() + stderr_thread.start() + + timed_out = False + try: + process.wait(timeout=timeout_seconds + 30) + except subprocess.TimeoutExpired: + timed_out = True + logger.error("GEAK v4 runner exceeded its hard timeout") + finally: + _terminate_process_group(process, logger) + stdout_thread.join(timeout=2) + stderr_thread.join(timeout=2) + + if timed_out: + raise TimeoutError(f"GEAK v4 timed out after {timeout_seconds} seconds") + + result = _read_json(result_path) + stderr_text = "".join(stderr) + if process.returncode != 0: + detail = result.get("error") if result else stderr_text[-4000:] + raise RuntimeError( + f"GEAK v4 runner failed with exit {process.returncode}: {detail}" + ) + if result is None: + raise RuntimeError(f"GEAK v4 runner did not write a valid result: {result_path}") + return "\n".join(part for part in ("".join(stdout), stderr_text) if part) + + +@register_agent("geak_v4") +def launch_agent( + eval_config: dict[str, Any], + task_config_dir: str, + workspace: str, +) -> str: + """Run GEAK v4 against the Arena workspace, applying its patch in place.""" + logger = logging.getLogger(__name__) + agent_config = _load_yaml(Path(__file__).with_name("agent_config.yaml")) + task_config = _load_yaml(task_config_dir) + workspace_path = Path(workspace).resolve() + if not workspace_path.is_dir(): + raise FileNotFoundError(f"Arena workspace does not exist: {workspace_path}") + + task_type = str(task_config.get("task_type") or "") + supported = {str(value) for value in agent_config.get("supported_task_types", [])} + if task_type not in supported: + raise ValueError( + f"GEAK v4 does not support task_type={task_type!r}; " + f"supported task types: {sorted(supported)}" + ) + sources = _declared_sources(task_config, workspace_path) + + workflow_dir = Path( + os.environ.get("GEAK_V4_WORKFLOW_DIR") + or agent_config.get("workflow_dir") + or "/opt/geak/kernel_workflow" + ).resolve() + workflow_script = workflow_dir / "kernel_workflow.js" + if not workflow_script.is_file(): + raise FileNotFoundError( + f"GEAK v4 workflow not found: {workflow_script}. " + "Point GEAK_V4_WORKFLOW_DIR at your GEAK kernel_workflow directory." + ) + claude_binary = shutil.which("claude") + if not claude_binary: + raise RuntimeError( + "Claude Code CLI ('claude') not found on PATH; install it and log in first." + ) + + # Run artifacts live OUTSIDE the workspace: GEAK copies kernel_path into its + # own exp tree, so nesting outputs under the workspace would recurse — and it + # keeps GEAK's scratch clear of the directory Arena scores. + run_id = ( + datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S") + + f"_{os.getpid()}_{uuid.uuid4().hex[:8]}" + ) + run_dir = workspace_path.parent / f".{workspace_path.name}_geak_v4" / run_id + eval_dir = run_dir / "eval" + exp_root = run_dir / "runs" + handoff_path = run_dir / "handoff.json" + result_path = run_dir / "result.json" + run_dir.mkdir(parents=True, exist_ok=True) + + prompt_builder = load_prompt_builder(AgentType.GEAK_V4, logger) + task_prompt = prompt_builder(task_config_dir, str(workspace_path), eval_config, logger) + if sources: + joined = ", ".join(f"`{name}`" for name in sources) + task_prompt += ( + "\n\n### GEAK/Arena Integration Contract\n" + "The task's config.yaml compile, correctness, and performance commands " + "are the measurement source of truth. Do not create, modify, or replace " + "any test, harness, config, reference, or timing file. Optimize only the " + f"declared source file(s): {joined}." + ) + + timeout_seconds = int(agent_config.get("timeout_seconds", 43200)) + handoff = { + "schema_version": 1, + "kernel_path": str(workspace_path), + "workflow_dir": str(workflow_dir), + "eval_dir": str(eval_dir), + "exp_root": str(exp_root), + "gpu_ids": _logical_gpu_ids(eval_config), + "budget": int(agent_config.get("budget", 6)), + "min_improve": float(agent_config.get("min_improve", 0.02)), + "deep_cost": int(agent_config.get("deep_cost", 2)), + "use_expert_skills": bool(agent_config.get("use_expert_skills", False)), + "task": task_prompt, + "model": str(agent_config.get("model", "claude-opus-4-8")), + "effort": str(agent_config.get("effort", "ultracode")), + "claude_cli_path": claude_binary, + "timeout_seconds": timeout_seconds, + "done_grace_seconds": float(agent_config.get("done_grace_seconds", 1800)), + "done_poll_seconds": float(agent_config.get("done_poll_seconds", 5)), + # GEAK's Director git-applies the validated patch straight into kernel_path + # (the Arena workspace). Arena's harness guard + independent re-score are + # the integrity boundary, so we let GEAK edit in place like forge does. + "apply_to_original": "true", + } + handoff_path.write_text( + json.dumps(handoff, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + logger.info("GEAK v4 preflight") + logger.info(" workflow: %s", workflow_script) + logger.info(" workspace (kernel_path): %s", workspace_path) + logger.info(" eval dir: %s", eval_dir) + logger.info(" optimize-only sources: %s", sources or "") + logger.info(" logical GPU IDs: %s", handoff["gpu_ids"]) + logger.info(" budget: %s timeout: %ss", handoff["budget"], timeout_seconds) + + output = _run_workflow_runner( + handoff_path, + result_path, + timeout_seconds=timeout_seconds, + logger=logger, + ) + result = _read_json(result_path) + if result is None: + raise RuntimeError(f"GEAK v4 runner did not write a valid result: {result_path}") + + status = str(result.get("status") or "unknown") + applied = str(result.get("applied_to_original", "unknown")).lower() == "true" + if status == "ok" and applied: + logger.info( + "GEAK v4 accepted a gain (speedup=%s); patch applied into the workspace", + result.get("final_speedup"), + ) + elif status in {"no_gain", "rejected"}: + logger.info( + "GEAK v4 produced no accepted gain (status=%s); workspace left at baseline", + status, + ) + else: + logger.warning( + "GEAK v4 finished with status=%s (%s)", + status, + result.get("reason") or "", + ) + return output + "\n" + json.dumps(result, sort_keys=True) diff --git a/agents/geak_v4/workflow_runner.py b/agents/geak_v4/workflow_runner.py new file mode 100644 index 00000000..d5da3faf --- /dev/null +++ b/agents/geak_v4/workflow_runner.py @@ -0,0 +1,876 @@ +#!/usr/bin/env python3 +"""Stable GEAK v4 kernel-workflow runner for external orchestrators. + +The GEAK JavaScript workflow can only execute inside Claude Code's dynamic +``Workflow`` runtime. This module keeps that volatile SDK/tool lifecycle out of +the Arena launcher: + +* map a versioned handoff onto ``kernel_workflow.js`` arguments; +* pin a known evaluation directory for completion/recovery; +* keep the SDK client alive when Workflow runs as a background task; +* recover the authoritative result from on-disk GEAK artifacts; and +* honor the handoff's ``apply_to_original`` (the Arena launcher sets ``"true"`` + so GEAK's Director applies the validated patch straight into the workspace). + +The command is intentionally usable in ``--dry-run`` mode without importing the +Claude Agent SDK or contacting a model. +""" + +from __future__ import annotations + +import argparse +import json +import math +import os +import shutil +import stat +import sys +import tempfile +import time +from pathlib import Path +from typing import Any, Iterable + + +SCHEMA_VERSION = 1 +ALLOWED_TOOLS = ["Workflow", "Bash", "Read", "Write"] +VALID_EFFORTS = {"low", "medium", "high", "xhigh", "max"} +DEFAULT_SETTINGS = {"enableWorkflows": True, "ultracode": True} +_JSON_SIZE_LIMIT = 8 * 1024 * 1024 +_SDK_OUTPUT_FILE_LIMIT = 8 * 1024 * 1024 +_TRANSCRIPT_SIZE_LIMIT = 8 * 1024 * 1024 +_TRANSCRIPT_JSON_LINE_LIMIT = 64 + + +class HandoffError(ValueError): + """The caller supplied an invalid GEAK handoff.""" + + +def _open_directory_fd( + path: Path | str, + *, + parent_fd: int | None = None, +) -> int: + flags = os.O_RDONLY + flags |= getattr(os, "O_CLOEXEC", 0) + flags |= getattr(os, "O_DIRECTORY", 0) + flags |= getattr(os, "O_NOFOLLOW", 0) + descriptor = os.open(path, flags, dir_fd=parent_fd) + metadata = os.fstat(descriptor) + if not stat.S_ISDIR(metadata.st_mode): + os.close(descriptor) + raise OSError(f"not a directory: {path}") + return descriptor + + +def _atomic_write_json( + path: Path, + payload: dict[str, Any], + *, + directory_fd: int | None = None, +) -> None: + """Write JSON without following attacker-created file symlinks. + + When ``directory_fd`` is supplied, the write stays pinned to that already + opened directory even if Workflow renames or replaces its pathname. + """ + encoded = ( + json.dumps(payload, indent=2, sort_keys=True) + "\n" + ).encode("utf-8") + owned_directory_fd = directory_fd is None + if directory_fd is None: + directory_fd = _open_directory_fd(path.parent) + + temporary_fd = -1 + temporary_name = "" + try: + proc_directory = Path(f"/proc/self/fd/{directory_fd}") + temporary_fd, temporary_path = tempfile.mkstemp( + prefix=f".{path.name}.tmp.", + dir=proc_directory, + ) + temporary_name = Path(temporary_path).name + metadata = os.fstat(temporary_fd) + if not stat.S_ISREG(metadata.st_mode) or metadata.st_nlink != 1: + raise OSError("atomic JSON temporary is not a private regular file") + with os.fdopen(temporary_fd, "wb", closefd=True) as stream: + temporary_fd = -1 + stream.write(encoded) + stream.flush() + os.fsync(stream.fileno()) + os.replace( + temporary_name, + path.name, + src_dir_fd=directory_fd, + dst_dir_fd=directory_fd, + ) + temporary_name = "" + os.fsync(directory_fd) + finally: + if temporary_fd >= 0: + os.close(temporary_fd) + if temporary_name: + try: + os.unlink(temporary_name, dir_fd=directory_fd) + except FileNotFoundError: + pass + if owned_directory_fd: + os.close(directory_fd) + + +def _read_bounded_text(path: Path, size_limit: int) -> str | None: + flags = os.O_RDONLY + flags |= getattr(os, "O_CLOEXEC", 0) + flags |= getattr(os, "O_NOFOLLOW", 0) + flags |= getattr(os, "O_NONBLOCK", 0) + try: + descriptor = os.open(path, flags) + except OSError: + return None + try: + metadata = os.fstat(descriptor) + if ( + not stat.S_ISREG(metadata.st_mode) + or metadata.st_size > size_limit + ): + return None + with os.fdopen(descriptor, "rb", closefd=False) as stream: + content = stream.read(size_limit + 1) + if len(content) > size_limit: + return None + return content.decode("utf-8") + except (OSError, UnicodeDecodeError): + return None + finally: + os.close(descriptor) + + +def _read_json(path: Path) -> dict[str, Any] | None: + content = _read_bounded_text(path, _JSON_SIZE_LIMIT) + if content is None: + return None + try: + value = json.loads(content) + except json.JSONDecodeError: + return None + return value if isinstance(value, dict) else None + + +def load_handoff(path: Path) -> dict[str, Any]: + value = _read_json(path) + if value is None: + raise HandoffError(f"handoff is not a readable JSON object: {path}") + if value.get("schema_version") != SCHEMA_VERSION: + raise HandoffError( + f"unsupported handoff schema_version={value.get('schema_version')!r}; " + f"expected {SCHEMA_VERSION}" + ) + return value + + +def _absolute_path(value: Any, field: str) -> Path: + if not isinstance(value, str) or not value.strip(): + raise HandoffError(f"{field} must be a non-empty absolute path") + path = Path(value) + if not path.is_absolute(): + raise HandoffError(f"{field} must be absolute: {path}") + return path.resolve() + + +def _positive_int(value: Any, field: str) -> int: + try: + parsed = int(value) + except (TypeError, ValueError) as exc: + raise HandoffError(f"{field} must be an integer") from exc + if parsed <= 0: + raise HandoffError(f"{field} must be positive") + return parsed + + +def _nonnegative_float(value: Any, field: str) -> float: + try: + parsed = float(value) + except (TypeError, ValueError) as exc: + raise HandoffError(f"{field} must be numeric") from exc + if not math.isfinite(parsed) or parsed < 0: + raise HandoffError(f"{field} must be finite and non-negative") + return parsed + + +def _gpu_ids(value: Any) -> str: + if isinstance(value, (list, tuple)): + value = ",".join(str(item) for item in value) + text = str(value if value is not None else "0") + parts = [part.strip() for part in text.split(",") if part.strip()] + if not parts or any(not part.isdigit() for part in parts): + raise HandoffError(f"gpu_ids must be comma-separated non-negative integers: {text!r}") + return ",".join(parts) + + +def map_workflow_args(handoff: dict[str, Any]) -> tuple[Path, dict[str, Any]]: + """Validate a handoff and return ``(script_path, workflow_args)``.""" + kernel_path = _absolute_path(handoff.get("kernel_path"), "kernel_path") + workflow_dir = _absolute_path(handoff.get("workflow_dir"), "workflow_dir") + eval_dir = _absolute_path(handoff.get("eval_dir"), "eval_dir") + exp_root = _absolute_path( + handoff.get("exp_root") or str(eval_dir.parent), + "exp_root", + ) + script_path = workflow_dir / "kernel_workflow.js" + + apply_to_original = str(handoff.get("apply_to_original", "false")).strip().lower() + if apply_to_original not in {"true", "false"}: + raise HandoffError( + "apply_to_original must be 'true' or 'false': " + f"{handoff.get('apply_to_original')!r}" + ) + + if not kernel_path.is_dir(): + raise HandoffError(f"kernel_path is not a directory: {kernel_path}") + if not script_path.is_file(): + raise HandoffError(f"GEAK kernel workflow not found: {script_path}") + if eval_dir.exists() and ( + not eval_dir.is_dir() or any(eval_dir.iterdir()) + ): + raise HandoffError(f"eval_dir must be absent or an empty directory: {eval_dir}") + for path, field in ((eval_dir, "eval_dir"), (exp_root, "exp_root")): + try: + path.relative_to(kernel_path) + except ValueError: + pass + else: + raise HandoffError( + f"{field} must not be inside kernel_path; GEAK copies kernel_path " + f"and would recursively copy its own outputs: {path}" + ) + + args: dict[str, Any] = { + "kernel_path": str(kernel_path), + "workflow_dir": str(workflow_dir), + "eval_dir": str(eval_dir), + "exp_root": str(exp_root), + "gpu_ids": _gpu_ids(handoff.get("gpu_ids", "0")), + "budget": _positive_int(handoff.get("budget", 6), "budget"), + "min_improve": _nonnegative_float( + handoff.get("min_improve", 0.02), + "min_improve", + ), + "deep_cost": _positive_int(handoff.get("deep_cost", 2), "deep_cost"), + "mode": "optimize", + # Handoff-driven: with "true" GEAK's Director git-applies the validated + # patch straight into kernel_path; with "false" the caller imports it. + "apply_to_original": apply_to_original, + } + task = handoff.get("task") + if task: + args["task"] = str(task) + if bool(handoff.get("use_expert_skills", False)): + args["use_expert_skills"] = "true" + return script_path, args + + +def build_prompt(script_path: Path, workflow_args: dict[str, Any]) -> str: + eval_dir = workflow_args["eval_dir"] + if workflow_args.get("apply_to_original") == "true": + patch_note = ( + "apply_to_original is true, so the Director writes the validated patch " + "back into kernel_path itself. " + ) + else: + patch_note = ( + "Do not edit the original kernel_path directly; apply_to_original is " + "false and the caller owns patch import. " + ) + return ( + "Invoke the Workflow tool exactly once with:\n" + f' scriptPath: "{script_path}"\n' + f" args: {json.dumps(workflow_args, ensure_ascii=False)}\n" + "Run the complete GEAK kernel pipeline through independent Director " + f"validation. {patch_note}When the " + "Workflow finishes, write its exact full return object as compact JSON to " + f'"{eval_dir}/workflow_return.json", then print exactly that compact JSON ' + "as the final line and print nothing after it." + ) + + +def _iter_message_text(message: Any) -> Iterable[str]: + """Yield text fragments from SDK objects across supported SDK shapes.""" + if message is None: + return + if isinstance(message, str): + if message.strip(): + yield message + return + if isinstance(message, dict): + for key in ("result", "text", "summary"): + value = message.get(key) + if isinstance(value, str) and value.strip(): + yield value + content = message.get("content") + if isinstance(content, str): + if content.strip(): + yield content + elif isinstance(content, (list, tuple)): + for item in content: + yield from _iter_message_text(item) + return + + for attribute in ("result", "text", "summary"): + value = getattr(message, attribute, None) + if isinstance(value, str) and value.strip(): + yield value + content = getattr(message, "content", None) + if isinstance(content, str): + if content.strip(): + yield content + elif isinstance(content, (list, tuple)): + for item in content: + yield from _iter_message_text(item) + + +def _finite_number(value: Any) -> bool: + try: + return math.isfinite(float(value)) + except (TypeError, ValueError): + return False + + +def _matches_pinned_patch(value: Any, eval_dir: Path) -> bool: + return isinstance(value, str) and value == str(eval_dir / "final_patch.diff") + + +def _valid_workflow_return( + value: Any, + eval_dir: Path, + *, + require_pinned_patch: bool = False, +) -> bool: + if not isinstance(value, dict): + return False + raw_eval_dir = value.get("eval_dir") + if not isinstance(raw_eval_dir, str): + return False + matches = raw_eval_dir == str(eval_dir) + status = value.get("validation_status") + patch_contract_ok = ( + not require_pinned_patch + or status not in {"accepted", "flagged"} + or _matches_pinned_patch(value.get("final_patch"), eval_dir) + ) + return ( + matches + and isinstance(status, str) + and _finite_number(value.get("final_geomean")) + and isinstance(value.get("final_patch"), str) + and patch_contract_ok + and ( + "workload_aligned" not in value + or isinstance(value.get("workload_aligned"), bool) + ) + ) + + +def _valid_director_validation( + value: Any, + eval_dir: Path | None = None, +) -> bool: + valid = ( + isinstance(value, dict) + and value.get("validation_status") in {"accepted", "flagged"} + and value.get("correctness") in {"pass", "fail"} + and _finite_number(value.get("director_verified_speedup_geomean")) + and value.get("applied_to_original") in {"true", "false"} + and isinstance(value.get("final_patch"), str) + ) + return valid and ( + eval_dir is None + or _matches_pinned_patch(value.get("final_patch"), eval_dir) + ) + + +def _terminal_artifact_exists(eval_dir: Path) -> bool: + workflow_return = _read_json(eval_dir / "workflow_return.json") + director_validation = _read_json(eval_dir / "director_validation.json") + return _valid_workflow_return( + workflow_return, + eval_dir, + require_pinned_patch=True, + ) or _valid_director_validation(director_validation, eval_dir) + + +def _extract_workflow_return(transcript: str, expected_eval_dir: Path) -> dict[str, Any] | None: + """Read the final compact JSON line without quadratic brace scanning.""" + lines = transcript.splitlines() + for raw_line in reversed(lines[-_TRANSCRIPT_JSON_LINE_LIMIT:]): + line = raw_line.strip() + if not line or len(line.encode("utf-8")) > _JSON_SIZE_LIMIT: + continue + try: + value = json.loads(line) + except json.JSONDecodeError: + continue + if _valid_workflow_return( + value, + expected_eval_dir, + require_pinned_patch=True, + ): + return value + return None + + +def _completed_producer_error( + state: dict[str, bool], + pending: set[str], + producer_error: list[BaseException], +) -> BaseException | None: + if not state["producer_done"]: + return None + if producer_error: + return producer_error[0] + if pending: + return RuntimeError( + "Claude SDK message stream ended with unfinished GEAK tasks: " + f"{sorted(pending)}" + ) + if not state["result_seen"]: + return RuntimeError( + "Claude SDK message stream ended without a ResultMessage or " + "a valid GEAK terminal artifact" + ) + return None + + +def invoke_via_sdk( + prompt: str, + *, + workflow_dir: Path, + eval_dir: Path, + model: str, + effort: str, + settings: str, + cli_path: str, + timeout_seconds: int, + done_grace_seconds: float, + done_poll_seconds: float, +) -> str: + """Invoke Claude Code while surviving synchronous and background Workflows.""" + try: + import anyio + from claude_agent_sdk import ClaudeAgentOptions, ClaudeSDKClient + except ImportError as exc: + raise RuntimeError( + "claude_agent_sdk is required for reliable GEAK Workflow lifecycle " + "handling; install it into the runner's Python (pip install " + "claude-agent-sdk)" + ) from exc + + option_extras: dict[str, Any] = {} + if effort in VALID_EFFORTS: + option_extras["effort"] = effort + if hasattr(os, "geteuid") and os.geteuid() == 0: + raise RuntimeError( + "GEAK v4 refuses to run Claude as root without a real OS sandbox; " + "use the Docker runner's non-root host UID mapping" + ) + sdk_env = { + "CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1", + } + + options = ClaudeAgentOptions( + model=model, + allowed_tools=ALLOWED_TOOLS, + permission_mode="bypassPermissions", + settings=settings, + extra_args=option_extras, + cwd=str(workflow_dir), + env=sdk_env, + **({"cli_path": cli_path} if cli_path else {}), + ) + + async def _run() -> str: + chunks: list[str] = [] + captured_chars = 0 + pending: set[str] = set() + state = { + "background_started": False, + "terminal_task_seen": False, + "result_seen": False, + "producer_done": False, + } + + with anyio.fail_after(timeout_seconds): + async with ClaudeSDKClient(options=options) as client: + await client.query(prompt) + producer_error: list[BaseException] = [] + + async def _receive() -> None: + nonlocal captured_chars + try: + async for message in client.receive_messages(): + for text in _iter_message_text(message): + remaining = _TRANSCRIPT_SIZE_LIMIT - captured_chars + if remaining > 0: + retained = text[:remaining] + chunks.append(retained) + captured_chars += len(retained) + compact = " ".join(text[:2000].split()) + if compact: + print( + f"[GEAK SDK] {compact[:500]}", + file=sys.stderr, + flush=True, + ) + + name = type(message).__name__ + if name == "TaskStartedMessage": + task_id = getattr(message, "task_id", None) + if task_id: + pending.add(str(task_id)) + state["background_started"] = True + elif name == "TaskNotificationMessage": + state["terminal_task_seen"] = True + task_id = getattr(message, "task_id", None) + if task_id: + pending.discard(str(task_id)) + output_file = getattr(message, "output_file", None) + if output_file: + output = _read_bounded_text( + Path(output_file), + _SDK_OUTPUT_FILE_LIMIT, + ) + remaining = _TRANSCRIPT_SIZE_LIMIT - captured_chars + if output and remaining > 0: + retained = output[:remaining] + chunks.append(retained) + captured_chars += len(retained) + elif name == "ResultMessage": + state["result_seen"] = True + except BaseException as exc: + producer_error.append(exc) + finally: + state["producer_done"] = True + + async with anyio.create_task_group() as task_group: + task_group.start_soon(_receive) + weak_deadline: float | None = None + while True: + # Match GEAK's lifecycle contract: a task notification + # is authoritative over an on-disk marker. The Director + # writes its JSON before StructuredOutput returns and + # the JS Workflow assembles its final result. + if pending and not state["producer_done"]: + await anyio.sleep(max(0.1, done_poll_seconds)) + continue + if _terminal_artifact_exists(eval_dir): + break + if state["result_seen"] and not state["background_started"]: + break + completion_error = _completed_producer_error( + state, + pending, + producer_error, + ) + if completion_error is not None: + raise completion_error + + weak_terminal = ( + state["background_started"] + and state["result_seen"] + and not pending + and ( + state["terminal_task_seen"] + or state["producer_done"] + ) + ) + if weak_terminal and weak_deadline is None: + weak_deadline = ( + time.monotonic() + max(0.0, done_grace_seconds) + ) + if weak_deadline is not None and time.monotonic() >= weak_deadline: + break + if ( + state["producer_done"] + and not state["background_started"] + and not state["result_seen"] + ): + break + await anyio.sleep(max(0.1, done_poll_seconds)) + task_group.cancel_scope.cancel() + return "\n".join(chunks)[:_TRANSCRIPT_SIZE_LIMIT] + + return anyio.run(_run) + + +def _number(value: Any) -> float | None: + try: + parsed = float(value) + except (TypeError, ValueError): + return None + return parsed if math.isfinite(parsed) else None + + +def normalize_result( + eval_dir: Path, + workflow_return: dict[str, Any] | None = None, +) -> dict[str, Any]: + """Build the stable runner result from GEAK's authoritative artifacts.""" + disk_return_path = eval_dir / "workflow_return.json" + disk_return = _read_json(disk_return_path) + disk_return_present = disk_return_path.exists() or disk_return_path.is_symlink() + disk_return_valid = _valid_workflow_return( + disk_return, + eval_dir, + require_pinned_patch=True, + ) + workflow_contract_invalid = disk_return_present and not disk_return_valid + if disk_return_valid: + workflow_return = disk_return + elif not _valid_workflow_return( + workflow_return, + eval_dir, + require_pinned_patch=True, + ): + workflow_return = {} + validation = _read_json(eval_dir / "director_validation.json") or {} + + validation_status = str( + validation.get("validation_status") + or workflow_return.get("validation_status") + or "unknown" + ).lower() + correctness = str(validation.get("correctness") or "unknown").lower() + workload_aligned = workflow_return.get("workload_aligned") is True + weighted = _number(validation.get("director_verified_speedup_weighted")) + geomean = _number(validation.get("director_verified_speedup_geomean")) + workflow_speedup = _number(workflow_return.get("final_speedup")) + speedup = ( + weighted + if workload_aligned and weighted is not None + else (geomean if geomean is not None else workflow_speedup) + ) + + patch_path = eval_dir / "final_patch.diff" + patch_exists = patch_path.is_file() and patch_path.stat().st_size > 0 + + accepted = validation_status == "accepted" and correctness == "pass" + gained = speedup is not None and speedup > 1.0 + director_valid = _valid_director_validation(validation, eval_dir) + primary_metric_valid = speedup is not None + if ( + accepted + and gained + and patch_exists + and director_valid + and not workflow_contract_invalid + ): + status = "ok" + elif accepted and director_valid and workflow_contract_invalid: + status = "error" + elif accepted and director_valid and not primary_metric_valid: + status = "error" + elif accepted and director_valid and not gained: + status = "no_gain" + elif validation_status == "flagged" or correctness == "fail": + status = "rejected" + else: + status = "error" + + if not director_valid and validation_status in {"accepted", "flagged"}: + reason = "GEAK Director artifact is missing or invalid" + elif workflow_contract_invalid: + reason = "GEAK workflow return artifact is present but invalid" + elif not accepted: + reason = ( + f"GEAK validation did not accept the candidate " + f"(status={validation_status}, correctness={correctness})" + ) + elif not primary_metric_valid: + metric = "weighted" if workload_aligned else "geomean" + reason = f"GEAK Director artifact has no finite {metric} speedup" + elif not gained: + reason = f"GEAK did not verify a speedup above 1.0x (speedup={speedup})" + elif not patch_exists: + reason = f"GEAK accepted a gain but produced no non-empty patch at {patch_path}" + else: + reason = "" + + return { + "schema_version": SCHEMA_VERSION, + "status": status, + "eval_dir": str(eval_dir), + "validation_status": validation_status, + "correctness": correctness, + "workload_aligned": workload_aligned, + "final_speedup": speedup, + "final_geomean": geomean, + "final_weighted": weighted, + "final_patch": str(patch_path), + "director_final_patch": validation.get("final_patch"), + "workflow_final_patch": workflow_return.get("final_patch"), + "report_path": str( + workflow_return.get("report_path") + or eval_dir / "tech_lead_report.md" + ), + "budget_used": workflow_return.get("budget_used"), + "budget_total": workflow_return.get("budget_total"), + "applied_to_original": validation.get("applied_to_original", "unknown"), + "reason": reason, + } + + +def run_handoff(handoff: dict[str, Any]) -> dict[str, Any]: + script_path, workflow_args = map_workflow_args(handoff) + eval_dir = Path(workflow_args["eval_dir"]) + eval_dir.parent.mkdir(parents=True, exist_ok=True) + prompt = build_prompt(script_path, workflow_args) + + timeout_seconds = _positive_int( + handoff.get("timeout_seconds", 43200), + "timeout_seconds", + ) + model = str(handoff.get("model") or "claude-opus-4-8") + effort = str(handoff.get("effort") or "ultracode") + settings_value = handoff.get("settings", DEFAULT_SETTINGS) + settings = ( + settings_value + if isinstance(settings_value, str) + else json.dumps(settings_value) + ) + cli_path = str( + handoff.get("claude_cli_path") + or os.environ.get("GEAK_CLAUDE_BIN") + or shutil.which("claude") + or "" + ).strip() + if not cli_path: + raise RuntimeError("Claude Code CLI not found; cannot run GEAK Workflow") + done_grace = _nonnegative_float( + handoff.get("done_grace_seconds", 1800), + "done_grace_seconds", + ) + done_poll = _nonnegative_float( + handoff.get("done_poll_seconds", 5), + "done_poll_seconds", + ) + + run_directory_fd = _open_directory_fd(eval_dir.parent) + try: + transcript = "" + invocation_error: Exception | None = None + try: + transcript = invoke_via_sdk( + prompt, + workflow_dir=script_path.parent, + eval_dir=eval_dir, + model=model, + effort=effort, + settings=settings, + cli_path=cli_path, + timeout_seconds=timeout_seconds, + done_grace_seconds=done_grace, + done_poll_seconds=done_poll, + ) + except Exception as exc: # disk recovery below may still prove completion + invocation_error = exc + + parsed_return = _extract_workflow_return(transcript, eval_dir) + if parsed_return: + eval_directory_fd = _open_directory_fd( + eval_dir.name, + parent_fd=run_directory_fd, + ) + try: + try: + os.stat( + "workflow_return.json", + dir_fd=eval_directory_fd, + follow_symlinks=False, + ) + except FileNotFoundError: + _atomic_write_json( + eval_dir / "workflow_return.json", + parsed_return, + directory_fd=eval_directory_fd, + ) + finally: + os.close(eval_directory_fd) + + if _terminal_artifact_exists(eval_dir): + result = normalize_result(eval_dir, parsed_return) + if invocation_error: + result["recovered_after_error"] = type(invocation_error).__name__ + return result + if invocation_error: + raise invocation_error + raise RuntimeError( + "GEAK Workflow exited without workflow_return.json or " + f"director_validation.json under {eval_dir}" + ) + finally: + os.close(run_directory_fd) + + +def _dry_run_result( + handoff: dict[str, Any], + script_path: Path, + workflow_args: dict[str, Any], +) -> dict[str, Any]: + return { + "schema_version": SCHEMA_VERSION, + "status": "dry_run", + "script_path": str(script_path), + "workflow_args": workflow_args, + "prompt": build_prompt(script_path, workflow_args), + "model": str(handoff.get("model") or "claude-opus-4-8"), + "effort": str(handoff.get("effort") or "ultracode"), + } + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="Run GEAK v4 kernel_workflow") + parser.add_argument("handoff", type=Path) + parser.add_argument("result", type=Path) + parser.add_argument( + "--dry-run", + action="store_true", + help="validate/map the handoff without importing the SDK or invoking Claude", + ) + namespace = parser.parse_args(argv) + + result_directory_fd = _open_directory_fd(namespace.result.parent) + try: + result: dict[str, Any] + try: + handoff = load_handoff(namespace.handoff) + if namespace.dry_run: + script_path, workflow_args = map_workflow_args(handoff) + result = _dry_run_result(handoff, script_path, workflow_args) + else: + result = run_handoff(handoff) + except Exception as exc: + result = { + "schema_version": SCHEMA_VERSION, + "status": "error", + "error_type": type(exc).__name__, + "error": str(exc), + } + _atomic_write_json( + namespace.result, + result, + directory_fd=result_directory_fd, + ) + print(json.dumps(result, ensure_ascii=False), flush=True) + return 1 + + _atomic_write_json( + namespace.result, + result, + directory_fd=result_directory_fd, + ) + print(json.dumps(result, ensure_ascii=False), flush=True) + return 1 if result.get("status") == "error" else 0 + finally: + os.close(result_directory_fd) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/example_configs/quickstart_geak_v4_mi300.yaml b/example_configs/quickstart_geak_v4_mi300.yaml new file mode 100644 index 00000000..fcc622c6 --- /dev/null +++ b/example_configs/quickstart_geak_v4_mi300.yaml @@ -0,0 +1,11 @@ +# Minimal first-run example for GEAK v4 on MI300/MI300X (gfx942). +# Complete agents/geak_v4/README.md setup before running it. +agent: + template: geak_v4 + +tasks: + - hip2hip/gpumode/GELU + +target_gpu_model: MI300 +log_directory: logs +workspace_directory_prefix: workspace diff --git a/src/module_registration.py b/src/module_registration.py index d6000820..01eb3b14 100755 --- a/src/module_registration.py +++ b/src/module_registration.py @@ -11,6 +11,7 @@ class AgentType(Enum): CLAUDE_CODE = "claude_code" CODEX = "codex" TASK_VALIDATOR = "task_validator" + GEAK_V4 = "geak_v4" GEAK_V3 = "geak_v3" GEAK_V3_TRITON = "geak_v3_triton" MINI_SWE_TRITON = "mini_swe_triton" @@ -66,6 +67,8 @@ def load_agent_launcher(agent_type: AgentType, logger: logging.Logger) -> Callab from agents.codex import launch_agent # noqa: F401 elif agent_type == AgentType.TASK_VALIDATOR: from agents.task_validator import launch_agent # noqa: F401 + elif agent_type == AgentType.GEAK_V4: + from agents.geak_v4 import launch_agent # noqa: F401 elif agent_type == AgentType.GEAK_V3: from agents.geak_v3 import launch_agent # noqa: F401 elif agent_type == AgentType.GEAK_V3_TRITON: @@ -109,7 +112,7 @@ def load_post_processing_handler(agent_type: AgentType, logger: logging.Logger) from agents.task_validator.validation_postprocessing import validation_post_processing logger.info(f"Using validation_post_processing for agent: {agent_name}") return validation_post_processing - elif agent_type in [AgentType.CURSOR, AgentType.CLAUDE_CODE, AgentType.CODEX, AgentType.GEAK_V3, AgentType.GEAK_V3_TRITON, AgentType.MINI_SWE_TRITON, AgentType.FORGE]: + elif agent_type in [AgentType.CURSOR, AgentType.CLAUDE_CODE, AgentType.CODEX, AgentType.GEAK_V4, AgentType.GEAK_V3, AgentType.GEAK_V3_TRITON, AgentType.MINI_SWE_TRITON, AgentType.FORGE]: logger.info(f"Using general_post_processing for agent: {agent_name}") return general_post_processing else: @@ -135,8 +138,8 @@ def load_prompt_builder(agent_type: AgentType, logger: logging.Logger) -> Callab agent_name = agent_type.value # Map agents to their prompt builder functions - if agent_type in [AgentType.CURSOR, AgentType.CLAUDE_CODE, AgentType.CODEX]: + if agent_type in [AgentType.CURSOR, AgentType.CLAUDE_CODE, AgentType.CODEX, AgentType.GEAK_V4]: logger.info(f"Using standard prompt_builder for agent: {agent_name}") return prompt_builder else: - raise NotImplementedError(f"Prompt builder not implemented for agent: {agent_name}") \ No newline at end of file + raise NotImplementedError(f"Prompt builder not implemented for agent: {agent_name}") diff --git a/tests/test_geak_v4.py b/tests/test_geak_v4.py new file mode 100644 index 00000000..7b793af9 --- /dev/null +++ b/tests/test_geak_v4.py @@ -0,0 +1,496 @@ +"""Offline tests for the GEAK v4 Arena adapter. + +These tests exercise the SDK-free surface: handoff mapping (including the +handoff-driven ``apply_to_original``), on-disk result recovery/normalization, GPU +namespace mapping, and the simplified launcher's handoff construction. They +intentionally do not invoke Claude, the Claude Agent SDK, GEAK, a container, or a +GPU. +""" + +from __future__ import annotations + +import importlib +import json +import logging +from pathlib import Path + +import pytest + +from agents.geak_v4 import workflow_runner +from src.module_registration import AgentType, load_agent_launcher + + +geak_launcher = importlib.import_module("agents.geak_v4.launch_agent") + + +def _write_json(path: Path, value: object) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(value) + "\n", encoding="utf-8") + + +def _handoff(tmp_path: Path) -> tuple[dict[str, object], Path, Path]: + kernel = tmp_path / "kernel" + workflow_dir = tmp_path / "geak" / "kernel_workflow" + eval_dir = tmp_path / "artifacts" / "eval" + kernel.mkdir(parents=True) + workflow_dir.mkdir(parents=True) + (workflow_dir / "kernel_workflow.js").write_text( + "// offline fixture\n", + encoding="utf-8", + ) + handoff: dict[str, object] = { + "schema_version": workflow_runner.SCHEMA_VERSION, + "kernel_path": str(kernel), + "workflow_dir": str(workflow_dir), + "eval_dir": str(eval_dir), + "exp_root": str(tmp_path / "artifacts" / "runs"), + "gpu_ids": "7", + "budget": 3, + "min_improve": 0.03, + "deep_cost": 1, + # The workflow always runs in optimize mode regardless of the handoff. + "mode": "author", + } + return handoff, kernel, eval_dir + + +def _workflow_return( + eval_dir: Path, + *, + workload_aligned: bool = False, +) -> dict[str, object]: + return { + "eval_dir": str(eval_dir), + "validation_status": "accepted", + "final_geomean": 1.20, + "final_speedup": 1.19, + "final_patch": str(eval_dir / "final_patch.diff"), + "workload_aligned": workload_aligned, + } + + +def _director_validation( + eval_dir: Path, + *, + validation_status: str = "accepted", + correctness: str = "pass", + geomean: object = 1.20, + weighted: object = 1.50, + applied_to_original: str = "true", +) -> dict[str, object]: + return { + "validation_status": validation_status, + "correctness": correctness, + "director_verified_speedup_geomean": geomean, + "director_verified_speedup_weighted": weighted, + "final_patch": str(eval_dir / "final_patch.diff"), + "applied_to_original": applied_to_original, + } + + +def _prepare_normalized_result( + tmp_path: Path, + *, + workload_aligned: bool = False, + validation_status: str = "accepted", + correctness: str = "pass", + geomean: object = 1.20, + weighted: object = 1.50, +) -> tuple[Path, dict[str, object]]: + eval_dir = tmp_path / "eval" + eval_dir.mkdir() + _write_json( + eval_dir / "workflow_return.json", + _workflow_return(eval_dir, workload_aligned=workload_aligned), + ) + _write_json( + eval_dir / "director_validation.json", + _director_validation( + eval_dir, + validation_status=validation_status, + correctness=correctness, + geomean=geomean, + weighted=weighted, + ), + ) + (eval_dir / "final_patch.diff").write_text( + "non-empty offline fixture\n", + encoding="utf-8", + ) + return eval_dir, workflow_runner.normalize_result(eval_dir) + + +def _make_task(tmp_path: Path) -> tuple[Path, Path]: + """Create a minimal task workspace + config.yaml with one kernel source.""" + workspace = tmp_path / "workspace" + source = workspace / "src" / "kernel.py" + source.parent.mkdir(parents=True) + source.write_text("value = 1\n", encoding="utf-8") + config = tmp_path / "config.yaml" + config.write_text( + "task_type: hip2hip\nsource_file_path:\n - src/kernel.py\n", + encoding="utf-8", + ) + return workspace, config + + +# --------------------------------------------------------------------------- # +# Registration + launcher +# --------------------------------------------------------------------------- # +def test_agent_registry_loads_geak_v4(): + assert AgentType.from_string("geak_v4") is AgentType.GEAK_V4 + assert ( + load_agent_launcher(AgentType.GEAK_V4, logging.getLogger(__name__)) + is geak_launcher.launch_agent + ) + + +def test_declared_sources_accepts_str_and_list(tmp_path): + workspace, _ = _make_task(tmp_path) + assert geak_launcher._declared_sources( + {"source_file_path": "src/kernel.py"}, workspace + ) == ["src/kernel.py"] + assert geak_launcher._declared_sources( + {"source_file_path": ["src/kernel.py"]}, workspace + ) == ["src/kernel.py"] + + +def test_declared_sources_empty_when_unset(tmp_path): + workspace, _ = _make_task(tmp_path) + assert geak_launcher._declared_sources({}, workspace) == [] + + +def test_declared_sources_fails_when_anchor_missing(tmp_path): + workspace, _ = _make_task(tmp_path) + with pytest.raises(FileNotFoundError, match="not found in workspace"): + geak_launcher._declared_sources( + {"source_file_path": ["src/missing.py"]}, workspace + ) + + +def test_launch_agent_rejects_unsupported_task_type(tmp_path): + workspace, _ = _make_task(tmp_path) + config = tmp_path / "bad.yaml" + config.write_text( + "task_type: repo2repo\nsource_file_path: [src/kernel.py]\n", + encoding="utf-8", + ) + with pytest.raises(ValueError, match="does not support task_type"): + geak_launcher.launch_agent({}, str(config), str(workspace)) + + +def test_launch_agent_writes_apply_in_place_handoff(tmp_path, monkeypatch): + workspace, config = _make_task(tmp_path) + workflow_dir = tmp_path / "geak" / "kernel_workflow" + workflow_dir.mkdir(parents=True) + (workflow_dir / "kernel_workflow.js").write_text("// fixture\n", encoding="utf-8") + + monkeypatch.setenv("GEAK_V4_WORKFLOW_DIR", str(workflow_dir)) + monkeypatch.setattr(geak_launcher.shutil, "which", lambda name: "/usr/bin/claude") + monkeypatch.setattr( + geak_launcher, + "load_prompt_builder", + lambda *args, **kwargs: (lambda *a, **k: "BASE PROMPT"), + ) + + captured: dict[str, object] = {} + + def fake_runner(handoff_path, result_path, *, timeout_seconds, logger): + handoff = json.loads(Path(handoff_path).read_text(encoding="utf-8")) + captured["handoff"] = handoff + Path(result_path).write_text( + json.dumps( + { + "schema_version": 1, + "status": "ok", + "applied_to_original": "true", + "final_speedup": 1.3, + "eval_dir": handoff["eval_dir"], + } + ) + + "\n", + encoding="utf-8", + ) + return "runner-output" + + monkeypatch.setattr(geak_launcher, "_run_workflow_runner", fake_runner) + + output = geak_launcher.launch_agent({"gpu_ids": "0"}, str(config), str(workspace)) + + handoff = captured["handoff"] + assert handoff["schema_version"] == 1 + # GEAK edits the workspace directly; Arena's harness guard re-scores it. + assert handoff["apply_to_original"] == "true" + assert handoff["kernel_path"] == str(workspace.resolve()) + assert handoff["workflow_dir"] == str(workflow_dir.resolve()) + assert handoff["claude_cli_path"] == "/usr/bin/claude" + assert "BASE PROMPT" in handoff["task"] + assert "src/kernel.py" in handoff["task"] + + # Artifacts must live OUTSIDE the scored workspace (hidden sibling dir). + eval_dir = Path(handoff["eval_dir"]).resolve() + assert not eval_dir.is_relative_to(workspace.resolve()) + artifact_root = workspace.parent / f".{workspace.name}_geak_v4" + assert eval_dir.parent.parent == artifact_root.resolve() + + assert "runner-output" in output + assert '"status": "ok"' in output + + +def test_launch_agent_errors_without_claude_cli(tmp_path, monkeypatch): + workspace, config = _make_task(tmp_path) + workflow_dir = tmp_path / "geak" / "kernel_workflow" + workflow_dir.mkdir(parents=True) + (workflow_dir / "kernel_workflow.js").write_text("// fixture\n", encoding="utf-8") + monkeypatch.setenv("GEAK_V4_WORKFLOW_DIR", str(workflow_dir)) + monkeypatch.setattr(geak_launcher.shutil, "which", lambda name: None) + + with pytest.raises(RuntimeError, match="Claude Code CLI"): + geak_launcher.launch_agent({}, str(config), str(workspace)) + + +def test_parallel_worker_maps_host_gpu_to_logical_zero(monkeypatch): + monkeypatch.setenv("AGENT_KERNEL_ARENA_HOST_GPU_ID", "7") + monkeypatch.setenv("ROCR_VISIBLE_DEVICES", "7") + monkeypatch.setenv("HIP_VISIBLE_DEVICES", "7") + monkeypatch.setenv("CUDA_VISIBLE_DEVICES", "7") + monkeypatch.setenv("GEAK_V4_GPU_IDS", "7") + + assert geak_launcher._logical_gpu_ids({"gpu_ids": "7"}) == "0" + + +# --------------------------------------------------------------------------- # +# workflow_runner: handoff mapping + apply_to_original +# --------------------------------------------------------------------------- # +def test_dry_run_defaults_apply_to_original_false(tmp_path, monkeypatch): + handoff, _, _ = _handoff(tmp_path) + handoff_path = tmp_path / "handoff.json" + result_path = tmp_path / "result.json" + _write_json(handoff_path, handoff) + + # A dry run must not enter the only function that imports claude_agent_sdk. + monkeypatch.setattr( + workflow_runner, + "invoke_via_sdk", + lambda *args, **kwargs: pytest.fail("dry-run invoked Claude SDK"), + ) + assert workflow_runner.main( + [str(handoff_path), str(result_path), "--dry-run"] + ) == 0 + + result = json.loads(result_path.read_text(encoding="utf-8")) + assert result["status"] == "dry_run" + assert result["workflow_args"]["mode"] == "optimize" + assert result["workflow_args"]["apply_to_original"] == "false" + assert result["workflow_args"]["gpu_ids"] == "7" + assert "Invoke the Workflow tool exactly once" in result["prompt"] + assert "apply_to_original is false" in result["prompt"] + + +def test_apply_to_original_is_handoff_driven(tmp_path): + handoff, _, _ = _handoff(tmp_path) + handoff["apply_to_original"] = "true" + + script_path, args = workflow_runner.map_workflow_args(handoff) + assert args["apply_to_original"] == "true" + assert args["mode"] == "optimize" + + prompt = workflow_runner.build_prompt(script_path, args) + assert "apply_to_original is true" in prompt + assert "the caller owns patch import" not in prompt + + +def test_invalid_apply_to_original_is_rejected(tmp_path): + handoff, _, _ = _handoff(tmp_path) + handoff["apply_to_original"] = "maybe" + with pytest.raises(workflow_runner.HandoffError, match="apply_to_original"): + workflow_runner.map_workflow_args(handoff) + + +@pytest.mark.parametrize("isolated_field", ["eval_dir", "exp_root"]) +def test_handoff_rejects_artifacts_inside_kernel(tmp_path, isolated_field): + handoff, kernel, _ = _handoff(tmp_path) + handoff[isolated_field] = str(kernel / "recursive-output") + + with pytest.raises( + workflow_runner.HandoffError, + match=rf"{isolated_field} must not be inside kernel_path", + ): + workflow_runner.map_workflow_args(handoff) + + +# --------------------------------------------------------------------------- # +# workflow_runner: on-disk readers + terminal artifact detection +# --------------------------------------------------------------------------- # +def test_runner_read_json_rejects_symlink_and_oversize(tmp_path, monkeypatch): + target = tmp_path / "target.json" + target.write_text('{"status": "ok"}\n', encoding="utf-8") + alias = tmp_path / "alias.json" + alias.symlink_to(target) + + assert workflow_runner._read_json(alias) is None + + monkeypatch.setattr(workflow_runner, "_JSON_SIZE_LIMIT", 4) + assert workflow_runner._read_json(target) is None + + +def test_runner_atomic_write_replaces_destination_symlink(tmp_path): + sentinel = tmp_path / "sentinel.txt" + sentinel.write_text("SAFE\n", encoding="utf-8") + runner_dir = tmp_path / "runner" + runner_dir.mkdir() + runner_result = runner_dir / "result.json" + runner_result.symlink_to(sentinel) + + workflow_runner._atomic_write_json(runner_result, {"status": "ok"}) + + assert sentinel.read_text(encoding="utf-8") == "SAFE\n" + assert not runner_result.is_symlink() + assert json.loads(runner_result.read_text(encoding="utf-8")) == {"status": "ok"} + + +def test_terminal_artifacts_require_complete_schema(tmp_path): + eval_dir = tmp_path / "eval" + eval_dir.mkdir() + _write_json( + eval_dir / "workflow_return.json", + { + "eval_dir": str(eval_dir), + "validation_status": "accepted", + "final_patch": "final_patch.diff", + }, + ) + _write_json( + eval_dir / "director_validation.json", + { + "validation_status": "accepted", + "correctness": "pass", + "final_patch": "final_patch.diff", + }, + ) + assert not workflow_runner._terminal_artifact_exists(eval_dir) + + _write_json( + eval_dir / "director_validation.json", + _director_validation(eval_dir), + ) + assert workflow_runner._terminal_artifact_exists(eval_dir) + + +def test_full_workflow_return_is_terminal_and_transcript_parser_ignores_noise( + tmp_path, +): + eval_dir = tmp_path / "eval" + eval_dir.mkdir() + complete = _workflow_return(eval_dir) + _write_json(eval_dir / "workflow_return.json", complete) + assert workflow_runner._terminal_artifact_exists(eval_dir) + + wrong_eval = dict(complete, eval_dir=str(tmp_path / "other")) + transcript = ( + 'setup={"enableWorkflows": true}\n' + + json.dumps(wrong_eval) + + '\npartial={"eval_dir": ' + + json.dumps(str(eval_dir)) + + "}\n" + + json.dumps(complete) + ) + assert workflow_runner._extract_workflow_return(transcript, eval_dir) == complete + + +def test_completed_background_stream_without_result_fails_immediately(): + state = { + "background_started": True, + "terminal_task_seen": True, + "result_seen": False, + "producer_done": True, + } + + error = workflow_runner._completed_producer_error(state, set(), []) + + assert isinstance(error, RuntimeError) + assert "without a ResultMessage" in str(error) + + +# --------------------------------------------------------------------------- # +# workflow_runner: result normalization +# --------------------------------------------------------------------------- # +def test_normalize_non_workload_uses_director_geomean(tmp_path): + _, result = _prepare_normalized_result( + tmp_path, + workload_aligned=False, + geomean=1.20, + weighted=9.0, + ) + + assert result["status"] == "ok" + assert result["final_speedup"] == pytest.approx(1.20) + assert result["final_geomean"] == pytest.approx(1.20) + assert result["final_weighted"] == pytest.approx(9.0) + assert result["applied_to_original"] == "true" + + +def test_normalize_workload_aligned_uses_director_weighted_speedup(tmp_path): + _, result = _prepare_normalized_result( + tmp_path, + workload_aligned=True, + geomean=1.20, + weighted=1.50, + ) + + assert result["status"] == "ok" + assert result["final_speedup"] == pytest.approx(1.50) + + +def test_normalize_rejects_non_finite_director_geomean(tmp_path): + _, result = _prepare_normalized_result( + tmp_path, + geomean=float("nan"), + weighted=1.50, + ) + + assert result["status"] == "error" + assert result["final_geomean"] is None + assert "missing or invalid" in result["reason"] + + +def test_normalize_workload_falls_back_when_weighted_is_non_finite(tmp_path): + _, result = _prepare_normalized_result( + tmp_path, + workload_aligned=True, + geomean=1.20, + weighted=float("nan"), + ) + + assert result["status"] == "ok" + assert result["final_speedup"] == pytest.approx(1.20) + assert result["final_weighted"] is None + + +def test_normalize_flagged_candidate_is_rejected(tmp_path): + _, result = _prepare_normalized_result( + tmp_path, + validation_status="flagged", + correctness="pass", + ) + + assert result["status"] == "rejected" + assert "did not accept" in result["reason"] + + +def test_normalize_rejects_patch_not_named_by_director(tmp_path): + eval_dir = tmp_path / "eval" + eval_dir.mkdir() + _write_json(eval_dir / "workflow_return.json", _workflow_return(eval_dir)) + validation = _director_validation(eval_dir) + validation["final_patch"] = str(eval_dir / "validated_elsewhere.diff") + _write_json(eval_dir / "director_validation.json", validation) + (eval_dir / "final_patch.diff").write_text( + "unvalidated patch\n", + encoding="utf-8", + ) + + result = workflow_runner.normalize_result(eval_dir) + + assert result["status"] == "error" + assert "Director artifact is missing or invalid" in result["reason"] From a75d9a1156c4c5d114943670ac19733b703638da Mon Sep 17 00:00:00 2001 From: yueliu14 Date: Wed, 29 Jul 2026 07:35:03 +0000 Subject: [PATCH 2/4] Wire geak_v4 into Docker runner and gate result on patch apply Addresses the two PR #73 review comments: P1: Map the geak_v4 template to Claude Code provisioning/checks in the Docker-first runner, mount+forward the host's GEAK_V4_WORKFLOW_DIR, and add a supported SDK setup step (make docker-setup-geak / container_setup_geak). The setup installs claude-agent-sdk into the persistent pip user-base via `pip install --target` (venv rejects --user, /opt/venv is unwritable by the non-root runner user) and forwards it on PYTHONPATH; it short-circuits when the image already ships the SDK. Also forward the host's Claude gateway auth (Core42/Primus-safe: ANTHROPIC_AUTH_TOKEN + ANTHROPIC_BASE_URL + model/TLS/ timeout vars) by name so the in-container CLI can authenticate. P2: normalize_result() gains require_applied; when the run requested apply_to_original, an accepted gain is only "ok" if applied_to_original == "true", otherwise it is an error (Arena would re-score the unmodified baseline). Covered by regression tests. Co-Authored-By: Claude Opus 4.8 --- Makefile | 8 +- agents/geak_v4/workflow_runner.py | 27 +++++- example_configs/quickstart_geak_v4_mi300.yaml | 3 +- src/scripts/docker_benchmark.sh | 92 ++++++++++++++++++- tests/test_docker_benchmark.sh | 66 +++++++++++++ tests/test_geak_v4.py | 41 +++++++++ 6 files changed, 232 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 3783387c..6e6e2392 100755 --- a/Makefile +++ b/Makefile @@ -6,7 +6,7 @@ SHELL := /bin/bash -.PHONY: help docker-shell docker-check-agents docker-smoke docker-run docker-parallel-run docker-setup-flydsl \ +.PHONY: help docker-shell docker-check-agents docker-smoke docker-run docker-parallel-run docker-setup-flydsl docker-setup-geak \ check-docker-runner check-evaluator check-held-out check-visualization \ visualization-build visualization-serve visualization-run \ sync-perf-helpers check-perf-helpers materialize-perf-workspace \ @@ -27,6 +27,7 @@ help: @echo " On other GPUs, pass a matching CONFIG explicitly" @echo " Images: gfx942->mi30x, gfx950->mi35x; override with AKA_DOCKER_IMAGE=..." @echo "make docker-setup-flydsl - Install FlyDSL when absent (for flydsl2flydsl, torch2flydsl, and triton2flydsl)" + @echo "make docker-setup-geak - Install the Claude Agent SDK when absent (for the geak_v4 agent)" @echo "make check-docker-runner - Check Docker runner syntax and runtime-specific arguments" @echo "make check-evaluator - Run centralized evaluator unit tests" @echo "make check-held-out - Run held-out module unit tests" @@ -74,6 +75,11 @@ docker-parallel-run: docker-setup-flydsl: @$(DOCKER_RUNNER) setup-flydsl +# Install the Claude Agent SDK into the container's persistent pip user-base when +# the selected image does not ship it. Needed by the geak_v4 agent. +docker-setup-geak: + @$(DOCKER_RUNNER) setup-geak + check-docker-runner: @bash tests/test_docker_benchmark.sh diff --git a/agents/geak_v4/workflow_runner.py b/agents/geak_v4/workflow_runner.py index d5da3faf..41dd4c19 100644 --- a/agents/geak_v4/workflow_runner.py +++ b/agents/geak_v4/workflow_runner.py @@ -611,8 +611,14 @@ def _number(value: Any) -> float | None: def normalize_result( eval_dir: Path, workflow_return: dict[str, Any] | None = None, + *, + require_applied: bool = False, ) -> dict[str, Any]: - """Build the stable runner result from GEAK's authoritative artifacts.""" + """Build the stable runner result from GEAK's authoritative artifacts. + + With ``require_applied`` set, an accepted gain is only ``"ok"`` when the + patch actually reached the workspace (``applied_to_original == "true"``). + """ disk_return_path = eval_dir / "workflow_return.json" disk_return = _read_json(disk_return_path) disk_return_present = disk_return_path.exists() or disk_return_path.is_symlink() @@ -651,15 +657,20 @@ def normalize_result( patch_path = eval_dir / "final_patch.diff" patch_exists = patch_path.is_file() and patch_path.stat().st_size > 0 + applied_to_original = str( + validation.get("applied_to_original", "unknown") + ).lower() accepted = validation_status == "accepted" and correctness == "pass" gained = speedup is not None and speedup > 1.0 director_valid = _valid_director_validation(validation, eval_dir) primary_metric_valid = speedup is not None + patch_applied_ok = (not require_applied) or applied_to_original == "true" if ( accepted and gained and patch_exists and director_valid + and patch_applied_ok and not workflow_contract_invalid ): status = "ok" @@ -667,6 +678,8 @@ def normalize_result( status = "error" elif accepted and director_valid and not primary_metric_valid: status = "error" + elif accepted and gained and director_valid and not patch_applied_ok: + status = "error" elif accepted and director_valid and not gained: status = "no_gain" elif validation_status == "flagged" or correctness == "fail": @@ -690,6 +703,12 @@ def normalize_result( reason = f"GEAK did not verify a speedup above 1.0x (speedup={speedup})" elif not patch_exists: reason = f"GEAK accepted a gain but produced no non-empty patch at {patch_path}" + elif not patch_applied_ok: + reason = ( + "GEAK accepted a gain but did not apply the patch to the workspace " + f"(applied_to_original={applied_to_original!r}); Arena would re-score " + "the unmodified baseline" + ) else: reason = "" @@ -795,7 +814,11 @@ def run_handoff(handoff: dict[str, Any]) -> dict[str, Any]: os.close(eval_directory_fd) if _terminal_artifact_exists(eval_dir): - result = normalize_result(eval_dir, parsed_return) + result = normalize_result( + eval_dir, + parsed_return, + require_applied=workflow_args.get("apply_to_original") == "true", + ) if invocation_error: result["recovered_after_error"] = type(invocation_error).__name__ return result diff --git a/example_configs/quickstart_geak_v4_mi300.yaml b/example_configs/quickstart_geak_v4_mi300.yaml index fcc622c6..caf6272a 100644 --- a/example_configs/quickstart_geak_v4_mi300.yaml +++ b/example_configs/quickstart_geak_v4_mi300.yaml @@ -1,5 +1,6 @@ # Minimal first-run example for GEAK v4 on MI300/MI300X (gfx942). -# Complete agents/geak_v4/README.md setup before running it. +# Complete agents/geak_v4/README.md setup first, and export GEAK_V4_WORKFLOW_DIR +# on the host (the Docker runner mounts/forwards it and installs the SDK). agent: template: geak_v4 diff --git a/src/scripts/docker_benchmark.sh b/src/scripts/docker_benchmark.sh index fe601cb3..7faa1117 100755 --- a/src/scripts/docker_benchmark.sh +++ b/src/scripts/docker_benchmark.sh @@ -311,6 +311,8 @@ resolve_required_agents() { claude|claude_code) printf 'claude_code\n' ;; cursor|cursor-agent) printf 'cursor\n' ;; codex) printf 'codex\n' ;; + # GEAK v4 drives Claude Code; extra deps handled in build/preflight. + geak_v4|geak-v4|geak) printf 'claude_code\n' ;; *) printf '%s\n' "$tmpl" ;; esac } @@ -327,7 +329,7 @@ normalize_check_agents() { all) normalized+=(codex claude_code cursor) ;; - claude|claude_code) + claude|claude_code|geak_v4|geak-v4|geak) normalized+=(claude_code) ;; cursor|cursor-agent) @@ -469,6 +471,10 @@ build_docker_args() { -e "TORCH_EXTENSIONS_DIR=/tmp/torch-extensions${cache_postfix}" -e "TRITON_CACHE_DIR=/tmp/triton-cache${cache_postfix}" -e "PYTHONUSERBASE=${CONTAINER_WORKDIR}/.aka-pyuserbase" + # geak_v4's claude-agent-sdk is installed with `pip install --target` into + # this host-mounted dir (see container_setup_geak); forward it on PYTHONPATH + # so the venv python can import it. Harmless when the dir is absent. + -e "PYTHONPATH=${CONTAINER_WORKDIR}/.aka-pyuserbase/geak-sdk" -e "MIOPEN_USER_DB_PATH=/tmp/miopen-cache${cache_postfix}" -e "MIOPEN_CACHE_DIR=/tmp/miopen-cache${cache_postfix}" -e "MIOPEN_CUSTOM_CACHE_DIR=/tmp/miopen-cache${cache_postfix}" @@ -509,6 +515,28 @@ build_docker_args() { if [[ "${AGENT_HOME_ISOLATION:-0}" == "1" ]]; then docker_args+=(-e "AGENT_KERNEL_ARENA_ISOLATED_HOME=1") fi + # Forward the host's Claude / Anthropic auth+config so agents that drive Claude + # (e.g. geak_v4) can authenticate without a persisted host login mounted at + # ~/.claude.json. This deliberately supports the AMD Core42 / Primus-safe + # gateway used on these hosts, where the credential is an ANTHROPIC_AUTH_TOKEN + # paired with an ANTHROPIC_BASE_URL (NOT a plain ANTHROPIC_API_KEY against + # api.anthropic.com). Each var is passed by name only (no "=value") so secrets + # stay out of argv / process listings, and only vars actually set on the host + # are forwarded. + local claude_env_var + for claude_env_var in \ + ANTHROPIC_AUTH_TOKEN ANTHROPIC_API_KEY ANTHROPIC_BASE_URL \ + ANTHROPIC_MODEL ANTHROPIC_DEFAULT_OPUS_MODEL \ + ANTHROPIC_DEFAULT_SONNET_MODEL ANTHROPIC_DEFAULT_HAIKU_MODEL \ + CLAUDE_CODE_SUBAGENT_MODEL API_TIMEOUT_MS \ + CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS \ + CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC \ + NODE_EXTRA_CA_CERTS SSL_CERT_FILE CURL_CA_BUNDLE \ + REQUESTS_CA_BUNDLE NODE_TLS_REJECT_UNAUTHORIZED; do + if [[ -n "${!claude_env_var:-}" ]]; then + docker_args+=(-e "$claude_env_var") + fi + done # GPU device nodes are group-owned (ROCm): /dev/dri/renderD* by `render` and # /dev/kfd by `render` or `video` depending on the host's udev rules. Add the @@ -537,6 +565,21 @@ build_docker_args() { mount_agent "$_agent" "$strict" done + # Mount the GEAK kernel_workflow checkout read-only at the same path and + # forward GEAK_V4_WORKFLOW_DIR so the launcher inside the container finds it. + if [[ -n "${GEAK_V4_WORKFLOW_DIR:-}" ]]; then + local geak_dir + geak_dir="$(cd "$GEAK_V4_WORKFLOW_DIR" 2>/dev/null && pwd || true)" + if [[ -n "$geak_dir" && -d "$geak_dir" ]]; then + add_mount "$geak_dir" "$geak_dir" ro + docker_args+=(-e "GEAK_V4_WORKFLOW_DIR=$geak_dir") + elif [[ "$strict" == "1" ]]; then + die "GEAK_V4_WORKFLOW_DIR is set but is not a directory: $GEAK_V4_WORKFLOW_DIR" + else + warn "GEAK_V4_WORKFLOW_DIR is set but is not a directory: $GEAK_V4_WORKFLOW_DIR; skipping GEAK mount" + fi + fi + # The base image lacks the GNU `time` binary and the container runs as a # non-root user (so it cannot apt-install it). Bind-mount the host binary # read-only so commands that invoke `/usr/bin/time` do not fail with 127. @@ -691,6 +734,11 @@ container_preflight() { container_smoke # Only verify the agent(s) this config actually uses (mounts are scoped the same way). container_check_agents $(resolve_required_agents "$config_name") + # GEAK v4 also needs the Claude Agent SDK and its kernel_workflow checkout. + if [[ "$(read_agent_template "$config_name")" == geak_v4 ]]; then + container_setup_geak + container_check_geak + fi python - "$config_name" <<'PY' import pathlib import sys @@ -723,6 +771,38 @@ container_setup_flydsl() { python -c 'import flydsl; print("flydsl=" + str(getattr(flydsl, "__version__", "unknown")) + " setup OK")' } +container_setup_geak() { + # Install claude-agent-sdk when the image does not ship it. + if python -c 'import claude_agent_sdk' 2>/dev/null; then + python -c 'import claude_agent_sdk; print("claude-agent-sdk already provided by image: " + str(getattr(claude_agent_sdk, "__version__", "unknown")) + "; nothing to install")' + return 0 + fi + # Install into a host-mounted target dir (survives the --rm container) and + # rely on the forwarded PYTHONPATH (see build_docker_args) to import it. + # `pip install --target` is the only option that works on the standard sglang + # runtimes: their python is a virtualenv rooted at /opt/venv, which both + # rejects `--user` (user-site disabled) and is unwritable by the non-root + # container user (so a plain in-venv install fails with EACCES). --target + # writes to a dir owned by the host UID and works on system-python images too. + # Trade-off: --target cannot see the venv's already-installed deps, so it + # pulls the SDK's full dependency closure (a few hundred MB, several minutes + # on first run). This is a one-time provisioning cost — later runs import the + # SDK via PYTHONPATH and short-circuit above. + local target="${PYTHONUSERBASE:-$PWD/.aka-pyuserbase}/geak-sdk" + echo "claude-agent-sdk not found in image; installing into $target ..." + python -m pip install --target "$target" claude-agent-sdk + PYTHONPATH="$target${PYTHONPATH:+:$PYTHONPATH}" python -c 'import claude_agent_sdk; print("claude-agent-sdk=" + str(getattr(claude_agent_sdk, "__version__", "unknown")) + " setup OK")' +} + +container_check_geak() { + # Confirm the kernel_workflow checkout is reachable inside the container. + local dir="${GEAK_V4_WORKFLOW_DIR:-/opt/geak/kernel_workflow}" + if [[ ! -f "$dir/kernel_workflow.js" ]]; then + die "GEAK kernel workflow not found: $dir/kernel_workflow.js. Export GEAK_V4_WORKFLOW_DIR on the host (the runner mounts and forwards it) to your GEAK kernel_workflow directory." + fi + echo "geak_workflow=$dir/kernel_workflow.js" +} + container_prepare_worker_home() { local state_root="${AGENT_STATE_MOUNT_ROOT:-/opt/aka-agent-state}" mkdir -p "$HOME" @@ -1015,9 +1095,19 @@ case "${1:-}" in AGENTS_STRICT=0 docker_exec 0 bash src/scripts/docker_benchmark.sh _container_setup_flydsl ;; + setup-geak) + select_runtime_for_host + REQUIRED_AGENTS="" + AGENTS_STRICT=0 + docker_exec 0 bash src/scripts/docker_benchmark.sh _container_setup_geak + ;; _container_setup_flydsl) container_setup_flydsl ;; + _container_setup_geak) + container_setup_geak + container_check_geak + ;; _container_smoke) container_smoke ;; diff --git a/tests/test_docker_benchmark.sh b/tests/test_docker_benchmark.sh index d68e2ffd..11f8a4b5 100755 --- a/tests/test_docker_benchmark.sh +++ b/tests/test_docker_benchmark.sh @@ -221,4 +221,70 @@ assert_has "claude_code" "${args[@]}" assert_has "cursor" "${args[@]}" assert_not_has "all" "${args[@]}" +# A geak_v4 config provisions the Claude Code CLI/auth and, when +# GEAK_V4_WORKFLOW_DIR is exported, mounts that checkout and forwards the var. +GEAK_HOME="$TEST_HOME/geak-home" +GEAK_PREFIX="$TEST_HOME/geak-node" +GEAK_CONFIG="$TEST_HOME/geak-config.yaml" +GEAK_WORKFLOW_DIR="$TEST_HOME/geak-checkout/kernel_workflow" +mkdir -p "$GEAK_HOME/.claude" "$GEAK_PREFIX/bin" "$GEAK_WORKFLOW_DIR" +touch \ + "$GEAK_HOME/.claude.json" \ + "$GEAK_PREFIX/bin/node" \ + "$GEAK_PREFIX/bin/claude" \ + "$GEAK_WORKFLOW_DIR/kernel_workflow.js" +printf 'agent:\n template: geak_v4\n' > "$GEAK_CONFIG" + +mapfile -t args < <(run_check_args \ + "$GEAK_HOME" \ + "$GEAK_CONFIG" \ + AKA_NODE_PREFIX="$GEAK_PREFIX" \ + GEAK_V4_WORKFLOW_DIR="$GEAK_WORKFLOW_DIR") +assert_has "$GEAK_PREFIX:/opt/claude-node:ro" "${args[@]}" +assert_has "$GEAK_HOME/.claude:$GEAK_HOME/.claude" "${args[@]}" +assert_has "$GEAK_HOME/.claude.json:$GEAK_HOME/.claude.json" "${args[@]}" +assert_has "claude_code" "${args[@]}" +assert_has "$GEAK_WORKFLOW_DIR:$GEAK_WORKFLOW_DIR:ro" "${args[@]}" +assert_has "GEAK_V4_WORKFLOW_DIR=$GEAK_WORKFLOW_DIR" "${args[@]}" +# The Claude Agent SDK is installed with `pip install --target` into the mounted +# user-base (setup-geak); its dir must be forwarded on PYTHONPATH so the venv +# python in the standard sglang images can import it. +assert_has "PYTHONPATH=/workspace/.aka-pyuserbase/geak-sdk" "${args[@]}" + +# Without GEAK_V4_WORKFLOW_DIR the checkout mount/env are absent. +mapfile -t args < <(run_check_args \ + "$GEAK_HOME" \ + "$GEAK_CONFIG" \ + AKA_NODE_PREFIX="$GEAK_PREFIX") +assert_has "claude_code" "${args[@]}" +assert_not_has "GEAK_V4_WORKFLOW_DIR=$GEAK_WORKFLOW_DIR" "${args[@]}" + +# The host's Claude gateway credentials are forwarded by name (value stays out of +# argv). These hosts use the AMD Core42 / Primus-safe gateway, where the credential +# is an ANTHROPIC_AUTH_TOKEN paired with an ANTHROPIC_BASE_URL. +mapfile -t args < <(run_check_args \ + "$GEAK_HOME" \ + "$GEAK_CONFIG" \ + AKA_NODE_PREFIX="$GEAK_PREFIX" \ + ANTHROPIC_AUTH_TOKEN=dummy-token-value \ + ANTHROPIC_BASE_URL=https://gateway.example/api) +assert_has "ANTHROPIC_AUTH_TOKEN" "${args[@]}" +assert_not_has "ANTHROPIC_AUTH_TOKEN=dummy-token-value" "${args[@]}" +assert_has "ANTHROPIC_BASE_URL" "${args[@]}" + +# A plain ANTHROPIC_API_KEY (e.g. api.anthropic.com auth) is likewise forwarded. +mapfile -t args < <(env -u ANTHROPIC_AUTH_TOKEN -u ANTHROPIC_BASE_URL \ + HOME="$GEAK_HOME" AKA_GPU_ARCH=gfx950 AKA_NODE_PREFIX="$GEAK_PREFIX" \ + ANTHROPIC_API_KEY=dummy-key-value \ + bash "$RUNNER" check-agents --config_name "$GEAK_CONFIG" 2>/dev/null) +assert_has "ANTHROPIC_API_KEY" "${args[@]}" +assert_not_has "ANTHROPIC_API_KEY=dummy-key-value" "${args[@]}" + +# When no Claude credentials are present on the host, none are forwarded. +mapfile -t args < <(env -u ANTHROPIC_AUTH_TOKEN -u ANTHROPIC_API_KEY -u ANTHROPIC_BASE_URL \ + HOME="$GEAK_HOME" AKA_GPU_ARCH=gfx950 AKA_NODE_PREFIX="$GEAK_PREFIX" \ + bash "$RUNNER" check-agents --config_name "$GEAK_CONFIG" 2>/dev/null) +assert_not_has "ANTHROPIC_AUTH_TOKEN" "${args[@]}" +assert_not_has "ANTHROPIC_API_KEY" "${args[@]}" + echo "PASS: docker_benchmark runtime and agent-selection argument tests" diff --git a/tests/test_geak_v4.py b/tests/test_geak_v4.py index 7b793af9..57e0a835 100644 --- a/tests/test_geak_v4.py +++ b/tests/test_geak_v4.py @@ -478,6 +478,47 @@ def test_normalize_flagged_candidate_is_rejected(tmp_path): assert "did not accept" in result["reason"] +def test_normalize_ok_requires_patch_applied_when_launcher_asks(tmp_path): + """An accepted gain that never reached the workspace must not report ok.""" + eval_dir = tmp_path / "eval" + eval_dir.mkdir() + _write_json(eval_dir / "workflow_return.json", _workflow_return(eval_dir)) + _write_json( + eval_dir / "director_validation.json", + _director_validation(eval_dir, applied_to_original="false"), + ) + (eval_dir / "final_patch.diff").write_text( + "non-empty offline fixture\n", + encoding="utf-8", + ) + + relaxed = workflow_runner.normalize_result(eval_dir) + assert relaxed["status"] == "ok" + + strict = workflow_runner.normalize_result(eval_dir, require_applied=True) + assert strict["status"] == "error" + assert strict["applied_to_original"] == "false" + assert "did not apply the patch to the workspace" in strict["reason"] + + +def test_normalize_ok_when_patch_applied_and_apply_required(tmp_path): + eval_dir = tmp_path / "eval" + eval_dir.mkdir() + _write_json(eval_dir / "workflow_return.json", _workflow_return(eval_dir)) + _write_json( + eval_dir / "director_validation.json", + _director_validation(eval_dir, applied_to_original="true"), + ) + (eval_dir / "final_patch.diff").write_text( + "non-empty offline fixture\n", + encoding="utf-8", + ) + + result = workflow_runner.normalize_result(eval_dir, require_applied=True) + assert result["status"] == "ok" + assert result["applied_to_original"] == "true" + + def test_normalize_rejects_patch_not_named_by_director(tmp_path): eval_dir = tmp_path / "eval" eval_dir.mkdir() From 45b68523e3ae501d2a659d4d9a17fc446ba98b31 Mon Sep 17 00:00:00 2001 From: Vincent Ouyang Date: Wed, 29 Jul 2026 19:10:06 +0000 Subject: [PATCH 3/4] Scope GEAK Docker credentials and dependencies --- src/scripts/docker_benchmark.sh | 89 ++++++++++++++++++++++----------- tests/test_docker_benchmark.sh | 68 +++++++++++++++++++++++-- 2 files changed, 125 insertions(+), 32 deletions(-) diff --git a/src/scripts/docker_benchmark.sh b/src/scripts/docker_benchmark.sh index 7faa1117..00ec90f7 100755 --- a/src/scripts/docker_benchmark.sh +++ b/src/scripts/docker_benchmark.sh @@ -13,6 +13,10 @@ SELECTED_GPU_ARCH="" SELECTED_IMAGE="" AGENT_STATE_MOUNT_ROOT="${AKA_AGENT_STATE_MOUNT_ROOT:-/opt/aka-agent-state}" DEFAULT_RUN_CONFIG="example_configs/quickstart_claude_mi300.yaml" +# Set by host-side commands after reading the selected run config. Keep this +# separate from REQUIRED_AGENTS because geak_v4 is normalized to claude_code +# before Docker arguments are built. +GEAK_V4_RUNTIME=0 # /opt/venv/bin is placed before /usr/local/bin and /usr/bin so that a bare # `python3` / `pytest` resolves to the torch-enabled venv interpreter rather than @@ -284,6 +288,24 @@ read_agent_template() { sed -nE 's/^[[:space:]]+template:[[:space:]]*["'"'"']?([A-Za-z0-9_]+).*/\1/p' "$config" | head -n 1 } +configure_geak_v4_runtime() { + local config="$1" + GEAK_V4_RUNTIME=0 + if [[ "$(read_agent_template "$config")" == "geak_v4" ]]; then + GEAK_V4_RUNTIME=1 + fi +} + +agent_list_contains() { + local agents="$1" + local expected="$2" + local agent + for agent in $agents; do + [[ "$agent" == "$expected" ]] && return 0 + done + return 1 +} + # task_validator delegates to a backend CLI; read which one. read_validator_backend() { local cfg="$HOST_ROOT/agents/task_validator/agent_config.yaml" @@ -471,10 +493,6 @@ build_docker_args() { -e "TORCH_EXTENSIONS_DIR=/tmp/torch-extensions${cache_postfix}" -e "TRITON_CACHE_DIR=/tmp/triton-cache${cache_postfix}" -e "PYTHONUSERBASE=${CONTAINER_WORKDIR}/.aka-pyuserbase" - # geak_v4's claude-agent-sdk is installed with `pip install --target` into - # this host-mounted dir (see container_setup_geak); forward it on PYTHONPATH - # so the venv python can import it. Harmless when the dir is absent. - -e "PYTHONPATH=${CONTAINER_WORKDIR}/.aka-pyuserbase/geak-sdk" -e "MIOPEN_USER_DB_PATH=/tmp/miopen-cache${cache_postfix}" -e "MIOPEN_CACHE_DIR=/tmp/miopen-cache${cache_postfix}" -e "MIOPEN_CUSTOM_CACHE_DIR=/tmp/miopen-cache${cache_postfix}" @@ -487,6 +505,14 @@ build_docker_args() { -w "$CONTAINER_WORKDIR" ) + # geak_v4's claude-agent-sdk is installed with `pip install --target` into + # this host-mounted dir (see container_setup_geak). Only put it on + # PYTHONPATH for GEAK runs so its dependency closure cannot shadow the + # runtime image's pinned packages for existing agents. + if [[ "$GEAK_V4_RUNTIME" == "1" ]]; then + docker_args+=(-e "PYTHONPATH=${CONTAINER_WORKDIR}/.aka-pyuserbase/geak-sdk") + fi + # The pinned gfx950 image ships root-owned AITER/FlyDSL caches, and its # /tmp/aiter_configs directory is not writable by the host UID used below. # Keep these overrides tied to that exact runtime so custom images and @@ -515,28 +541,28 @@ build_docker_args() { if [[ "${AGENT_HOME_ISOLATION:-0}" == "1" ]]; then docker_args+=(-e "AGENT_KERNEL_ARENA_ISOLATED_HOME=1") fi - # Forward the host's Claude / Anthropic auth+config so agents that drive Claude - # (e.g. geak_v4) can authenticate without a persisted host login mounted at - # ~/.claude.json. This deliberately supports the AMD Core42 / Primus-safe - # gateway used on these hosts, where the credential is an ANTHROPIC_AUTH_TOKEN - # paired with an ANTHROPIC_BASE_URL (NOT a plain ANTHROPIC_API_KEY against - # api.anthropic.com). Each var is passed by name only (no "=value") so secrets - # stay out of argv / process listings, and only vars actually set on the host - # are forwarded. - local claude_env_var - for claude_env_var in \ - ANTHROPIC_AUTH_TOKEN ANTHROPIC_API_KEY ANTHROPIC_BASE_URL \ - ANTHROPIC_MODEL ANTHROPIC_DEFAULT_OPUS_MODEL \ - ANTHROPIC_DEFAULT_SONNET_MODEL ANTHROPIC_DEFAULT_HAIKU_MODEL \ - CLAUDE_CODE_SUBAGENT_MODEL API_TIMEOUT_MS \ - CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS \ - CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC \ - NODE_EXTRA_CA_CERTS SSL_CERT_FILE CURL_CA_BUNDLE \ - REQUESTS_CA_BUNDLE NODE_TLS_REJECT_UNAUTHORIZED; do - if [[ -n "${!claude_env_var:-}" ]]; then - docker_args+=(-e "$claude_env_var") - fi - done + # Forward the host's Claude / Anthropic auth+config only for GEAK execution + # containers that provision Claude Code. Requiring both conditions keeps + # existing Claude/task-validator runs unchanged and prevents setup-geak + # (which has no agent CLI) from receiving runtime credentials. Each var is + # passed by name only (no "=value") so secrets stay out of argv / process + # listings. + if [[ "$GEAK_V4_RUNTIME" == "1" ]] && agent_list_contains "$agents" claude_code; then + local claude_env_var + for claude_env_var in \ + ANTHROPIC_AUTH_TOKEN ANTHROPIC_API_KEY ANTHROPIC_BASE_URL \ + ANTHROPIC_MODEL ANTHROPIC_DEFAULT_OPUS_MODEL \ + ANTHROPIC_DEFAULT_SONNET_MODEL ANTHROPIC_DEFAULT_HAIKU_MODEL \ + CLAUDE_CODE_SUBAGENT_MODEL API_TIMEOUT_MS \ + CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS \ + CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC \ + NODE_EXTRA_CA_CERTS SSL_CERT_FILE CURL_CA_BUNDLE \ + REQUESTS_CA_BUNDLE NODE_TLS_REJECT_UNAUTHORIZED; do + if [[ -n "${!claude_env_var:-}" ]]; then + docker_args+=(-e "$claude_env_var") + fi + done + fi # GPU device nodes are group-owned (ROCm): /dev/dri/renderD* by `render` and # /dev/kfd by `render` or `video` depending on the host's udev rules. Add the @@ -565,9 +591,9 @@ build_docker_args() { mount_agent "$_agent" "$strict" done - # Mount the GEAK kernel_workflow checkout read-only at the same path and - # forward GEAK_V4_WORKFLOW_DIR so the launcher inside the container finds it. - if [[ -n "${GEAK_V4_WORKFLOW_DIR:-}" ]]; then + # Mount the GEAK kernel_workflow checkout only for GEAK runs so an exported + # host setting does not change the container surface for existing agents. + if [[ "$GEAK_V4_RUNTIME" == "1" && -n "${GEAK_V4_WORKFLOW_DIR:-}" ]]; then local geak_dir geak_dir="$(cd "$GEAK_V4_WORKFLOW_DIR" 2>/dev/null && pwd || true)" if [[ -n "$geak_dir" && -d "$geak_dir" ]]; then @@ -961,6 +987,7 @@ run_parallel() { local config_name config_name="$(extract_config_name "$@")" select_runtime_for_config "$config_name" + configure_geak_v4_runtime "$config_name" REQUIRED_AGENTS="$(resolve_required_agents "$config_name")" AGENTS_STRICT=1 @@ -1041,6 +1068,7 @@ case "${1:-}" in shift config_name="$(extract_config_name "$@")" select_runtime_for_config "$config_name" + configure_geak_v4_runtime "$config_name" # Only the configured agent's CLI/auth is required for a run. REQUIRED_AGENTS="$(resolve_required_agents "$config_name")" AGENTS_STRICT=1 @@ -1055,6 +1083,7 @@ case "${1:-}" in shift config_name="$(extract_config_name "$@")" select_runtime_for_config "$config_name" + configure_geak_v4_runtime "$config_name" REQUIRED_AGENTS="$(resolve_required_agents "$config_name")" AGENTS_STRICT=1 docker_exec 0 bash src/scripts/docker_benchmark.sh _container_preflight "$config_name" @@ -1075,6 +1104,7 @@ case "${1:-}" in if [[ -z "${AKA_AGENTS:-}" ]]; then [[ -f "$config_name" ]] || die "config file not found: $config_name" fi + configure_geak_v4_runtime "$config_name" # By default, check only the CLI selected by CONFIG. AKA_AGENTS can # request one, several, or `all` explicitly. REQUIRED_AGENTS="$(normalize_check_agents "$(resolve_required_agents "$config_name")")" @@ -1097,6 +1127,7 @@ case "${1:-}" in ;; setup-geak) select_runtime_for_host + GEAK_V4_RUNTIME=1 REQUIRED_AGENTS="" AGENTS_STRICT=0 docker_exec 0 bash src/scripts/docker_benchmark.sh _container_setup_geak diff --git a/tests/test_docker_benchmark.sh b/tests/test_docker_benchmark.sh index 11f8a4b5..95eeaf1b 100755 --- a/tests/test_docker_benchmark.sh +++ b/tests/test_docker_benchmark.sh @@ -73,6 +73,10 @@ assert_cache_args_absent() { TEST_HOME="$(mktemp -d)" trap 'rm -rf "$TEST_HOME"' EXIT +UNRELATED_GEAK_WORKFLOW_DIR="$TEST_HOME/unrelated-geak-workflow" +GEAK_SDK_PYTHONPATH="PYTHONPATH=/workspace/.aka-pyuserbase/geak-sdk" +mkdir -p "$UNRELATED_GEAK_WORKFLOW_DIR" +touch "$UNRELATED_GEAK_WORKFLOW_DIR/kernel_workflow.js" bash -n "$RUNNER" @@ -125,7 +129,12 @@ mkdir -p \ touch "$CURSOR_HOME/.local/bin/cursor-agent" printf 'agent:\n template: cursor\n' > "$CURSOR_CONFIG" -mapfile -t args < <(run_check_args "$CURSOR_HOME" "$CURSOR_CONFIG") +mapfile -t args < <(run_check_args \ + "$CURSOR_HOME" \ + "$CURSOR_CONFIG" \ + ANTHROPIC_AUTH_TOKEN=cursor-must-not-receive-this \ + ANTHROPIC_BASE_URL=https://gateway.example/api \ + GEAK_V4_WORKFLOW_DIR="$UNRELATED_GEAK_WORKFLOW_DIR") assert_has "$CURSOR_HOME/.local/share/cursor-agent:$CURSOR_HOME/.local/share/cursor-agent:ro" "${args[@]}" assert_has "$CURSOR_HOME/.cursor:$CURSOR_HOME/.cursor" "${args[@]}" assert_has "$CURSOR_HOME/.config/cursor:$CURSOR_HOME/.config/cursor" "${args[@]}" @@ -133,6 +142,34 @@ assert_has "_container_check_agents" "${args[@]}" assert_has "cursor" "${args[@]}" assert_not_has "$CURSOR_HOME/.claude:$CURSOR_HOME/.claude" "${args[@]}" assert_not_has "$CURSOR_HOME/.codex:$CURSOR_HOME/.codex" "${args[@]}" +assert_not_has "ANTHROPIC_AUTH_TOKEN" "${args[@]}" +assert_not_has "ANTHROPIC_BASE_URL" "${args[@]}" +assert_not_has "$GEAK_SDK_PYTHONPATH" "${args[@]}" +assert_not_has "$UNRELATED_GEAK_WORKFLOW_DIR:$UNRELATED_GEAK_WORKFLOW_DIR:ro" "${args[@]}" +assert_not_has "GEAK_V4_WORKFLOW_DIR=$UNRELATED_GEAK_WORKFLOW_DIR" "${args[@]}" + +# A Codex-only config likewise receives neither Claude credentials nor GEAK's +# dependency path/mount, even when both are configured on the host. +CODEX_HOME="$TEST_HOME/codex-home" +CODEX_PREFIX="$TEST_HOME/codex-node" +CODEX_CONFIG="$TEST_HOME/codex-config.yaml" +mkdir -p "$CODEX_HOME/.codex" "$CODEX_PREFIX/bin" +touch "$CODEX_PREFIX/bin/node" "$CODEX_PREFIX/bin/codex" +printf 'agent:\n template: codex\n' > "$CODEX_CONFIG" + +mapfile -t args < <(run_check_args \ + "$CODEX_HOME" \ + "$CODEX_CONFIG" \ + AKA_NODE_PREFIX="$CODEX_PREFIX" \ + ANTHROPIC_API_KEY=codex-must-not-receive-this \ + GEAK_V4_WORKFLOW_DIR="$UNRELATED_GEAK_WORKFLOW_DIR") +assert_has "$CODEX_PREFIX:/opt/node:ro" "${args[@]}" +assert_has "$CODEX_HOME/.codex:$CODEX_HOME/.codex" "${args[@]}" +assert_has "codex" "${args[@]}" +assert_not_has "ANTHROPIC_API_KEY" "${args[@]}" +assert_not_has "$GEAK_SDK_PYTHONPATH" "${args[@]}" +assert_not_has "$UNRELATED_GEAK_WORKFLOW_DIR:$UNRELATED_GEAK_WORKFLOW_DIR:ro" "${args[@]}" +assert_not_has "GEAK_V4_WORKFLOW_DIR=$UNRELATED_GEAK_WORKFLOW_DIR" "${args[@]}" # A natively installed Claude CLI is a launcher in ~/.local/bin that resolves # into ~/.local/share/claude/versions. Both sides of that symlink must be @@ -149,12 +186,20 @@ touch \ ln -s ../share/claude/versions/2.1.0 "$NATIVE_CLAUDE_HOME/.local/bin/claude" printf 'agent:\n template: claude_code\n' > "$NATIVE_CLAUDE_CONFIG" -mapfile -t args < <(run_check_args "$NATIVE_CLAUDE_HOME" "$NATIVE_CLAUDE_CONFIG") +mapfile -t args < <(run_check_args \ + "$NATIVE_CLAUDE_HOME" \ + "$NATIVE_CLAUDE_CONFIG" \ + ANTHROPIC_AUTH_TOKEN=claude-must-not-receive-this \ + GEAK_V4_WORKFLOW_DIR="$UNRELATED_GEAK_WORKFLOW_DIR") assert_has "$NATIVE_CLAUDE_HOME/.local/bin:$NATIVE_CLAUDE_HOME/.local/bin:ro" "${args[@]}" assert_has "$NATIVE_CLAUDE_HOME/.local/share/claude:$NATIVE_CLAUDE_HOME/.local/share/claude:ro" "${args[@]}" assert_has "$NATIVE_CLAUDE_HOME/.claude:$NATIVE_CLAUDE_HOME/.claude" "${args[@]}" assert_has "$NATIVE_CLAUDE_HOME/.claude.json:$NATIVE_CLAUDE_HOME/.claude.json" "${args[@]}" assert_has "claude_code" "${args[@]}" +assert_not_has "ANTHROPIC_AUTH_TOKEN" "${args[@]}" +assert_not_has "$GEAK_SDK_PYTHONPATH" "${args[@]}" +assert_not_has "$UNRELATED_GEAK_WORKFLOW_DIR:$UNRELATED_GEAK_WORKFLOW_DIR:ro" "${args[@]}" +assert_not_has "GEAK_V4_WORKFLOW_DIR=$UNRELATED_GEAK_WORKFLOW_DIR" "${args[@]}" # Omitting --config_name uses the one-task MI300/MI300X Claude quickstart. mapfile -t args < <( @@ -249,7 +294,23 @@ assert_has "GEAK_V4_WORKFLOW_DIR=$GEAK_WORKFLOW_DIR" "${args[@]}" # The Claude Agent SDK is installed with `pip install --target` into the mounted # user-base (setup-geak); its dir must be forwarded on PYTHONPATH so the venv # python in the standard sglang images can import it. -assert_has "PYTHONPATH=/workspace/.aka-pyuserbase/geak-sdk" "${args[@]}" +assert_has "$GEAK_SDK_PYTHONPATH" "${args[@]}" + +# The explicit setup command has no run config or required agent CLI, but still +# needs the GEAK-only dependency path and workflow mount for its container check. +mapfile -t args < <( + env \ + HOME="$TEST_HOME" \ + AKA_GPU_ARCH=gfx950 \ + GEAK_V4_WORKFLOW_DIR="$GEAK_WORKFLOW_DIR" \ + ANTHROPIC_AUTH_TOKEN=setup-must-not-receive-this \ + bash "$RUNNER" setup-geak 2>/dev/null +) +assert_has "$GEAK_SDK_PYTHONPATH" "${args[@]}" +assert_has "$GEAK_WORKFLOW_DIR:$GEAK_WORKFLOW_DIR:ro" "${args[@]}" +assert_has "GEAK_V4_WORKFLOW_DIR=$GEAK_WORKFLOW_DIR" "${args[@]}" +assert_has "_container_setup_geak" "${args[@]}" +assert_not_has "ANTHROPIC_AUTH_TOKEN" "${args[@]}" # Without GEAK_V4_WORKFLOW_DIR the checkout mount/env are absent. mapfile -t args < <(run_check_args \ @@ -258,6 +319,7 @@ mapfile -t args < <(run_check_args \ AKA_NODE_PREFIX="$GEAK_PREFIX") assert_has "claude_code" "${args[@]}" assert_not_has "GEAK_V4_WORKFLOW_DIR=$GEAK_WORKFLOW_DIR" "${args[@]}" +assert_has "$GEAK_SDK_PYTHONPATH" "${args[@]}" # The host's Claude gateway credentials are forwarded by name (value stays out of # argv). These hosts use the AMD Core42 / Primus-safe gateway, where the credential From d1ab8f70d340572b4e131deb20114f9f9cd531e2 Mon Sep 17 00:00:00 2001 From: Peter Moutsias Date: Thu, 23 Jul 2026 13:55:56 -0400 Subject: [PATCH 4/4] fix: pin docs dependencies to Python 3.10 compatible versions RTD builds with Python 3.10; recompiled requirements.txt so transitive pins (ipython, sphinx, myst-parser) resolve to 3.10-compatible versions. --- docs/sphinx/requirements.txt | 39 +++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/docs/sphinx/requirements.txt b/docs/sphinx/requirements.txt index 77b63b2d..bdb7d7c6 100644 --- a/docs/sphinx/requirements.txt +++ b/docs/sphinx/requirements.txt @@ -1,3 +1,5 @@ +# This file was autogenerated by uv via the following command: +# uv pip compile docs/sphinx/requirements.in --python-version 3.10 --output-file docs/sphinx/requirements.txt accessible-pygments==0.0.5 # via pydata-sphinx-theme alabaster==1.0.0 @@ -37,11 +39,13 @@ debugpy==1.8.21 # via ipykernel decorator==5.3.1 # via ipython -docutils==0.22.4 +docutils==0.21.2 # via # myst-parser # pydata-sphinx-theme # sphinx +exceptiongroup==1.3.1 + # via ipython executing==2.2.1 # via stack-data fastjsonschema==2.21.2 @@ -64,12 +68,10 @@ importlib-metadata==9.0.0 # myst-nb ipykernel==7.3.0 # via myst-nb -ipython==9.15.0 +ipython==8.39.0 # via # ipykernel # myst-nb -ipython-pygments-lexers==1.1.1 - # via ipython jedi==0.20.0 # via ipython jinja2==3.1.6 @@ -92,7 +94,7 @@ jupyter-core==5.9.1 # jupyter-client # nbclient # nbformat -markdown-it-py==4.2.0 +markdown-it-py==3.0.0 # via # mdit-py-plugins # myst-parser @@ -108,7 +110,7 @@ mdurl==0.1.2 # via markdown-it-py myst-nb==1.4.0 # via rocm-docs-core -myst-parser==5.1.0 +myst-parser==4.0.1 # via myst-nb nbclient==0.11.0 # via @@ -135,9 +137,7 @@ platformdirs==4.11.0 prompt-toolkit==3.0.52 # via ipython psutil==7.2.2 - # via - # ipykernel - # ipython + # via ipykernel ptyprocess==0.7.0 # via pexpect pure-eval==0.2.3 @@ -154,10 +154,9 @@ pygments==2.20.0 # via # accessible-pygments # ipython - # ipython-pygments-lexers # pydata-sphinx-theme # sphinx -pyjwt[crypto]==2.13.0 +pyjwt==2.13.0 # via pygithub pynacl==1.6.2 # via pygithub @@ -184,10 +183,8 @@ requests==2.34.2 # pygithub # sphinx rocm-docs-core @ git+https://github.com/ROCm/rocm-docs-core.git@develop - # via -r /home/pmoutsia/AgentKernelArena/docs/sphinx/requirements.in -roman-numerals==4.1.0 - # via sphinx -rpds-py==2026.6.3 + # via -r docs/sphinx/requirements.in +rpds-py==0.30.0 # via # jsonschema # referencing @@ -199,7 +196,7 @@ snowballstemmer==3.1.1 # via sphinx soupsieve==2.9.1 # via beautifulsoup4 -sphinx==9.1.0 +sphinx==8.1.3 # via # breathe # myst-nb @@ -217,7 +214,7 @@ sphinx-book-theme==1.1.4 # via rocm-docs-core sphinx-copybutton==0.5.2 # via rocm-docs-core -sphinx-design==0.7.0 +sphinx-design==0.6.1 # via rocm-docs-core sphinx-external-toc==1.1.0 # via rocm-docs-core @@ -234,7 +231,7 @@ sphinxcontrib-htmlhelp==2.1.0 sphinxcontrib-jsmath==1.0.1 # via sphinx sphinxcontrib-mermaid==1.0.0 - # via -r /home/pmoutsia/AgentKernelArena/docs/sphinx/requirements.in + # via -r docs/sphinx/requirements.in sphinxcontrib-qthelp==2.0.0 # via sphinx sphinxcontrib-serializinghtml==2.0.0 @@ -245,6 +242,8 @@ stack-data==0.6.3 # via ipython tabulate==0.10.0 # via jupyter-cache +tomli==2.4.1 + # via sphinx tornado==6.5.7 # via # ipykernel @@ -261,10 +260,14 @@ traitlets==5.15.1 typing-extensions==4.16.0 # via # beautifulsoup4 + # cryptography + # exceptiongroup + # ipython # jupyter-client # myst-nb # pydata-sphinx-theme # pygithub + # pyjwt # referencing # sqlalchemy urllib3==2.7.0