Agent Builder Lab

A hands-on experiment project that implements and compares agent builders across AWS Bedrock, Google ADK, Azure AI, OpenAI Agents SDK, and Anthropic Claude — using identical scenarios on each platform.

Why This Project

AI agent frameworks are proliferating rapidly. But when you give them the same tasks, the implementation approach, performance, and cost differ significantly across platforms. This project was built to verify those differences directly in code, beyond what marketing docs can tell you.

Platforms

Platform	SDK	Default Model
AWS Bedrock	boto3 (Converse API)	Claude Sonnet 4
Google ADK	google-adk	Gemini 2.5 Flash
Azure AI	azure-ai-projects	GPT-5.4
OpenAI Agents	openai-agents	GPT-5.4
Anthropic Claude	anthropic	Claude Sonnet 4.6

Test Scenarios (4)

All scenarios are implemented identically across every platform.

#	Scenario	Key Test Points
1	RAG + Tool Use	Can it combine document retrieval with external API calls?
2	Multi-Agent Workflow	Pipeline where agents collaborate: researcher → writer → reviewer
3	Customer Service Bot	Multi-turn dialogue, sentiment awareness, escalation policy compliance
4	Data Analysis Agent	CSV data analysis → insight generation → markdown report

Quick Start

1. Install

# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Full install (all platform SDKs)
pip install -e ".[dev]"

# Or install specific platforms only
pip install -e ".[openai,anthropic]"

2. Configure API Keys

cp .env.example .env

Open .env and fill in the keys for the platforms you want to use:

# Only fill in the platforms you need
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...

3. Run a Scenario

# Specific platform + scenario
lab-run --scenario rag_tool_use --platform openai_agents

# Full comparison across all platforms
lab-run --scenario all --platform all

# Using Makefile
make run SCENARIO=rag_tool_use PLATFORM=anthropic_claude

4. Run a Module Directly

python -m src.platforms.openai_agents.rag_tool_use

5. Tests

make test
# or
pytest -v

Project Structure

agent-builder-lab/
├── config/
│   ├── settings.yaml              # Per-platform model and parameter settings
│   └── prompts/                   # Shared prompts per scenario (for fair comparison)
│       ├── rag_tool_use.yaml
│       ├── multi_agent.yaml
│       ├── customer_service.yaml
│       └── data_analysis.yaml
│
├── data/
│   ├── documents/                 # Source documents for RAG
│   ├── datasets/                  # CSV files for data analysis
│   └── test_inputs/               # Standard test queries per scenario (JSON)
│
├── src/
│   ├── common/                    # Shared modules
│   │   ├── base_agent.py          # AgentProtocol, AgentRequest, AgentResponse
│   │   ├── base_tool.py           # ToolProtocol, ToolResult
│   │   ├── types.py               # Platform, Scenario enums
│   │   ├── config.py              # .env + YAML config loader
│   │   ├── tools/                 # Shared tools (web_search, calculator)
│   │   └── utils/                 # Logging, timing utilities
│   │
│   ├── platforms/                 # Per-platform implementations
│   │   ├── aws_bedrock/           # client.py + 4 scenarios
│   │   ├── google_adk/
│   │   ├── azure_ai/
│   │   ├── openai_agents/
│   │   └── anthropic_claude/
│   │
│   └── evaluation/                # Evaluation framework
│       ├── runner.py              # CLI runner (lab-run command)
│       ├── metrics.py             # Latency, token, cost measurement
│       ├── comparator.py          # Cross-platform comparison tables
│       └── report.py             # Markdown report generation
│
├── notebooks/                     # Per-scenario comparison notebooks
├── docs/                          # Platform notes and comparison results
└── tests/

Design

Protocol-Based Common Interface

All platform implementations follow the same AgentProtocol. Using typing.Protocol instead of ABC avoids forcing each SDK's unique style while normalizing results into AgentResponse for comparison.

class AgentProtocol(Protocol):
    async def run(self, request: AgentRequest) -> AgentResponse: ...
    async def reset(self) -> None: ...

Fair Comparison Structure

• Identical prompts — scenario prompts managed in one place under config/prompts/
• Identical test inputs — standard query sets under data/test_inputs/
• Identical metrics — latency, token usage, cost estimate, response quality

Per-Platform client.py

Each platform directory has a client.py that handles SDK initialization and authentication, allowing scenario files to focus purely on business logic.

Implementation Order

The project skeleton and shared modules are complete. Implement scenarios in the following order:

Step 1: Complete the first scenario on the most familiar platform

src/platforms/openai_agents/rag_tool_use.py   ← recommended starting point

Once RAG + Tool Use works end-to-end on one platform, the rest become pattern-translation exercises.

Step 2: Extend to other platforms

openai_agents/rag_tool_use.py  ✅ done
  → anthropic_claude/rag_tool_use.py
  → aws_bedrock/rag_tool_use.py
  → google_adk/rag_tool_use.py
  → azure_ai/rag_tool_use.py

Step 3: Move to the next scenario

Scenario 1: RAG + Tool Use      ✅
Scenario 2: Multi-Agent Workflow  ← next
Scenario 3: Customer Service Bot
Scenario 4: Data Analysis Agent

Step 4: Comparative analysis

# Run all and check the report
lab-run --scenario all --platform all

Use the Jupyter notebooks in notebooks/ for detailed per-scenario comparisons, and summarize findings in docs/comparison-summary.md.

Configuration

Change Model

Edit config/settings.yaml to modify the model ID and parameters per platform:

platforms:
  openai_agents:
    model_id: "gpt-5.4"         # swap to a different model
    max_tokens: 4096
    temperature: 0.7

Edit Prompts

Modify system prompts and evaluation criteria per scenario in config/prompts/*.yaml.

Add Test Inputs

Add test queries to data/test_inputs/*.json — they are automatically included in evaluation.

Commands

Command	Description
make dev	Install all dependencies (dev mode)
make test	Run tests
make lint	Lint check
make format	Auto-format code
make run SCENARIO=... PLATFORM=...	Run a scenario
make run-all	Run full comparison
make clean	Clean build artifacts

Requirements

• Python 3.11+
• API keys / credentials for the platforms you want to test