Android Agent-S2 Setup and Running Guide

This guide will help you set up and run the Android Agent-S2 system for automated Android UI interaction tasks with comprehensive episode scoring and logging.

📚 References

This project builds upon two key frameworks:

• Android World: A dynamic benchmarking environment for Android UI automation tasks
• Agent-S: A hierarchical agent architecture for complex task execution

For more information about the underlying frameworks, please refer to their respective repositories and documentation.

📋 Prerequisites

1. System Requirements

• Operating System: Windows, macOS, or Linux
• RAM: At least 8GB (16GB recommended)
• Storage: At least 10GB free space
• Python: 3.11+ (required for android_world compatibility)

2. Required Software

• Android Studio: Latest version
• Conda: For Python environment management
• Git: For cloning repositories

🚀 Step-by-Step Setup

Step 1: Clone the Repository

# Clone the Agent-S repository
git clone 
cd Agent-S

# Initialize and update submodules
git submodule update --init --recursive

Step 2: Set Up Android Emulator

2.1 Install Android Studio

1. Download Android Studio from https://developer.android.com/studio
2. Install Android Studio following the installation wizard
3. Open Android Studio and complete the initial setup

2.2 Create Android Virtual Device (AVD)

1. Open Android Studio
2. Go to Tools → AVD Manager
3. Click Create Virtual Device
4. Select Pixel 6 for hardware
5. Choose Tiramisu, API Level 33 for System Image
6. Name the AVD: AndroidWorldAvd
7. Click Finish

Note: Watch the setup video for detailed visual instructions.

2.3 Launch Emulator from Command Line

Important: Launch the emulator from command line, NOT from Android Studio UI.

# Find your Android SDK path (usually in Android Studio settings)
# Common paths:
# Windows: C:\Users\username\AppData\Local\Android\Sdk
# macOS: ~/Library/Android/sdk
# Linux: ~/Android/Sdk

# Set ANDROID_HOME environment variable
export ANDROID_HOME=/path/to/your/Android/Sdk

# Launch the emulator with required flags
$ANDROID_HOME/emulator/emulator -avd AndroidWorldAvd -grpc 8554

Critical: The -grpc 8554 flag is required for communication with the accessibility forwarding app.

Step 3: Set Up Python Environment

# Create conda environment with current Python version
conda create -n android_world python=3.11

# Activate the environment
conda activate android_world

# Verify Python version
python --version  # Should show Python 3.11.x

Step 4: Install Dependencies

4.1 Install Base Requirements

# Navigate to the Agent-S directory
cd Agent-S

# Install base requirements
pip install -r requirements.txt

4.2 Install Android World Dependencies

# Navigate to android_world directory
cd android_world

# Install android_world dependencies
pip install -e .

4.3 Install Additional Dependencies

# Install additional packages needed for Android Agent-S2
pip install absl-py
pip install pillow
pip install opencv-python
pip install requests
pip install tiktoken

Step 5: Install All Requirements

# Make sure you're in the android_world conda environment
conda activate android_world

# Install all requirements (including Android Agent-S2 dependencies)
pip install -r requirements.txt

Step 6: Set Up Environment Variables

# Set OpenAI API key (required for LLM operations)
export OPENAI_API_KEY="your-openai-api-key-here"

# Set Android SDK path
export ANDROID_HOME="/path/to/your/Android/Sdk"

# Add platform-tools to PATH
export PATH=$PATH:$ANDROID_HOME/platform-tools

🎯 Running the Android Agent-S2

Prerequisites Check

Before running, ensure:

1. ✅ Android emulator is running with -grpc 8554 flag
2. ✅ Conda environment android_world is activated
3. ✅ OpenAI API key is set
4. ✅ All dependencies are installed

Available Run Scripts

The system provides several run scripts for different use cases:

1. Basic Task Execution (run_task.py)

# Navigate to the s2android directory
cd Agent-S/gui_agents/s2android

# Run with default settings (ContactsAddContact task)
python run_task.py

# Run with specific model
python run_task.py --model=gpt-4o-mini

# Run with specific task
python run_task.py --task=ContactsAddContact

# Run with custom API key
python run_task.py --api_key=your-api-key

2. Task with Episode Scoring (run_task_with_scorer.py) ⭐ Recommended

# Run with comprehensive episode scoring and metrics
python run_task_with_scorer.py

# Run with specific task and model
python run_task_with_scorer.py --task=ContactsAddContact --model=gpt-4o-mini

# Run with custom parameters
python run_task_with_scorer.py --task=SystemWifiTurnOn --model=gpt-4o-mini --console_port=5554

3. Batch Task Execution (run_all_tasks_with_scorer.py)

# Run all available tasks with scoring
python run_all_tasks_with_scorer.py

# Run with specific model and max steps
python run_all_tasks_with_scorer.py --model=gpt-4o-mini --max_steps=20

# Start from a specific task (useful for resuming)
python run_all_tasks_with_scorer.py --start_from_task=ContactsAddContact

# Exclude specific tasks
python run_all_tasks_with_scorer.py --exclude_tasks=ContactsDeleteContact,SystemWifiTurnOn

4. Screenshot Testing (run_simple_screenshot_test.py)

# Test screenshot capture and processing
python run_simple_screenshot_test.py

# Test with specific parameters
python run_simple_screenshot_test.py --model=gpt-4o-mini

5. Screenshot Comparison (run_screenshot_comparison.py)

# Compare screenshots between different runs
python run_screenshot_comparison.py

# Compare with specific parameters
python run_screenshot_comparison.py --model=gpt-4o-mini

Advanced Usage

# Run with custom ADB path
python run_task_with_scorer.py --adb_path=/path/to/adb

# Run with different console port
python run_task_with_scorer.py --console_port=5556

# Run with emulator setup (only needed once)
python run_task_with_scorer.py --perform_emulator_setup=True

# Run with specific task and model
python run_task_with_scorer.py --task=ContactsAddContact --model=gpt-4o-mini

📊 Episode Scoring System

The Android Agent-S2 includes a comprehensive episode scoring system that evaluates agent performance across multiple dimensions:

Scoring Components

The episode scorer computes a 0-100 score based on:

• Goal Completion: +50 points (primary success metric)
• Subtask Coverage: Up to +20 points (planning effectiveness)
• Step Efficiency: Up to +10 points (minimal steps)
• Action Validity: Up to +5 points (correct actions)
• Verifier Alignment: Up to +5 points (verification success)

Penalties

• Replan Penalty: Up to -10 points (excessive replanning)
• Stall Penalty: Up to -10 points (lack of progress)
• Cost Penalty: Up to -5 points (token usage)
• Latency Penalty: Up to -5 points (response time)

Metrics Tracked

• Steps taken: Number of execution steps
• Actions performed: Total actions and state-changing actions
• Token usage: Input/output tokens for planner, executor, and DAG
• Latency: Response times for each component
• Verification: Success rate of action verification
• Planning: Number of planning attempts and subtask completion

📁 Logging and Data Storage

Log Directory Structure

All logs are stored in gui_agents/s2android/logs/ with timestamped filenames:

logs/
├── qa_logs_YYYYMMDD_HHMMSS.json          # Question-Answer logs
├── metrics_YYYYMMDD_HHMMSS.jsonl         # Episode metrics
├── qa_logs_TaskName_YYYYMMDD_HHMMSS.json # Task-specific QA logs
└── metrics_TaskName_YYYYMMDD_HHMMSS.jsonl # Task-specific metrics

Log File Types

1. QA Logs (qa_logs_*.json)

Contains detailed question-answer interactions between the agent and verifier:

• Task goal: The original task objective
• Task success: Whether the task was completed successfully
• QA pairs: All question-answer interactions during execution
• Verification results: Success/failure of each action verification
• Reasoning: Detailed reasoning for each decision

2. Metrics Logs (metrics_*.jsonl)

Contains comprehensive episode metrics in JSONL format:

• Final score: Overall episode score (0-100)
• Success status: Whether the goal was achieved
• Component scores: Breakdown of scoring components
• Performance metrics: Steps, actions, tokens, latency
• Planning metrics: Subtasks planned and completed

Accessing Logs

# View recent QA logs
ls -la gui_agents/s2android/logs/qa_logs_*.json | tail -5

# View recent metrics
ls -la gui_agents/s2android/logs/metrics_*.jsonl | tail -5

# Analyze specific task logs
grep "ContactsAddContact" gui_agents/s2android/logs/metrics_*.jsonl

# View log contents
cat gui_agents/s2android/logs/qa_logs_20250811_143108.json

Log Analysis

The logs can be analyzed to:

• Track performance trends across multiple runs
• Identify failure patterns and improvement opportunities
• Compare different models and configurations
• Debug agent behavior and decision-making
• Generate performance reports and visualizations

🔧 Troubleshooting

Common Issues

1. ADB Not Found

# Error: adb not found in common Android SDK paths
# Solution: Set correct ANDROID_HOME and PATH
export ANDROID_HOME="/path/to/your/Android/Sdk"
export PATH=$PATH:$ANDROID_HOME/platform-tools

2. Emulator Not Running

# Check if emulator is running
adb devices

# If no devices listed, restart emulator with correct flags
$ANDROID_HOME/emulator/emulator -avd AndroidWorldAvd -grpc 8554

3. Import Errors

# If you get android_world import errors
cd Agent-S/android_world
pip install -e .

4. Python Version Issues

# Ensure you're using Python 3.11+
python --version

# If not, recreate conda environment
conda deactivate
conda remove -n android_world --all
conda create -n android_world python=3.11
conda activate android_world

Debug Mode

# Run with verbose logging
python run_task_with_scorer.py --verbosity=1

# Check emulator status
adb devices
adb shell getprop ro.build.version.release

📊 Expected Output

When running successfully, you should see:

🚀 Starting Android Agent-S2 Contact Task
📱 Using device on port: 5554
🤖 Using model: gpt-4o-mini
🔧 Setting up Android environment...
✅ Successfully imported android_world components
📋 Loading task registry...
🎯 Using default task: ContactsAddContact
🎯 Task Goal: Add a new contact with name 'John Doe' and phone number '555-1234'
🤖 Creating Android Agent-S2...
🔄 Starting task execution...
==================================================
🎯 Goal: Add a new contact with name 'John Doe' and phone number '555-1234'
==================================================

📝 Step 1/20
------------------------------
Worker → Add contact name: Enter the contact's first and last name
🔧 WORKER EXECUTION (Step 1):
   Subtask: Add contact name: Enter the contact's first and last name
   Plan: I need to add a contact name. Let me first open the contacts app...
   ✅ Screen changed detected
🔄 Action: {"action_type": "open_app", "app_name": "Contacts"}

📊 Task Results
==================================================
🎯 Goal: Add a new contact with name 'John Doe' and phone number '555-1234'
📈 Success Score: 1.0/1.0
✅ Agent Completed: Yes
🏆 Result: SUCCESS ✅

🏅 Episode Score: 85.2 / 100
   🎯 Goal: ✅ SUCCESS
   📊 Components:
      Goal: +50.0
      Coverage: +18.5
      Efficiency: +8.2
      Validity: +4.8
      Verifier: +4.7
      Penalties: -1.0 (replans), -0.0 (stalls), -0.0 (cost), -0.0 (latency)
   📈 Stats: 12 steps, 1 replans, 15 actions, 8 state changes

🎯 Available Tasks

The system supports various Android tasks:

Contact Management

• ContactsAddContact: Add a new contact with name and phone number
• ContactsEditContact: Edit an existing contact
• ContactsDeleteContact: Delete a contact
• ContactsNewContactDraft: Create a contact draft

System Settings

• SystemWifiTurnOn: Turn on WiFi
• SystemWifiTurnOff: Turn off WiFi
• SystemBluetoothTurnOn: Turn on Bluetooth
• SystemBluetoothTurnOff: Turn off Bluetooth

File Management

• FilesMoveFile: Move a file to a different location
• FilesDeleteFile: Delete a file
• FilesCreateFolder: Create a new folder

Note Taking (Markor)

• MarkorAddNoteHeader: Add a header to a note
• MarkorChangeNoteContent: Modify note content
• MarkorCreateFolder: Create a new folder in Markor

Expense Tracking

• ExpenseAddSingle: Add a single expense
• ExpenseAddMultiple: Add multiple expenses
• ExpenseDeleteSingle: Delete a single expense
• ExpenseDeleteMultiple: Delete multiple expenses
• ExpenseDeleteDuplicates: Remove duplicate expenses

🔍 Monitoring and Debugging

Check Emulator Status

# List connected devices
adb devices

# Check emulator properties
adb shell getprop

# Check running processes
adb shell ps

View Logs

# View system logs
adb logcat

# View specific app logs
adb logcat | grep "com.android.contacts"

Debug Agent Behavior

# Run with detailed logging
python run_task_with_scorer.py --verbosity=2

# Check episode data collection
# Look for episode_images_* directories after running

📁 Project Structure

Agent-S/
├── gui_agents/
│   └── s2android/
│       ├── agents/                    # Agent implementations
│       │   ├── agent_s.py            # Main Android Agent-S2
│       │   ├── worker.py             # Worker agent
│       │   ├── manager.py            # Manager agent
│       │   ├── verifier_agent.py     # Verification agent
│       │   └── grounding.py          # Grounding utilities
│       ├── core/                     # Core modules
│       ├── memory/                   # Procedural memory
│       ├── utils/                    # Utilities
│       │   └── episode_scorer.py     # Episode scoring system
│       ├── logs/                     # Generated logs
│       ├── run_task.py               # Basic task execution
│       ├── run_task_with_scorer.py   # Task with scoring ⭐
│       ├── run_all_tasks_with_scorer.py # Batch execution
│       ├── run_simple_screenshot_test.py # Screenshot testing
│       └── run_screenshot_comparison.py # Screenshot comparison
├── android_world/                    # Android World framework
├── requirements.txt                  # All dependencies (including Android Agent-S2)
└── README.md                       # This file

🤝 Support

If you encounter issues:

1. Check the troubleshooting section above
2. Verify all prerequisites are met
3. Check emulator is running with correct flags
4. Ensure conda environment is activated
5. Verify API key is set correctly
6. Review logs in gui_agents/s2android/logs/ for detailed error information

For additional support, please check the main project README or create an issue in the repository.

📝 Notes

• The emulator must be launched with -grpc 8554 flag for proper communication
• Python 3.11+ is required for android_world compatibility
• The system saves comprehensive logs for analysis in gui_agents/s2android/logs/
• Episode scoring provides detailed performance metrics (0-100 scale)
• All runs generate both QA logs and metrics logs for complete analysis
• The verifier agent provides detailed analysis of agent performance
• Field identification has been enhanced with hint_text analysis

📚 Additional Resources

Framework Documentation

• Android World GitHub: Official repository with setup instructions, task definitions, and API documentation
• Agent-S GitHub: Official repository with agent architecture details and implementation guides

Research Papers

• Android World: "Android World: A Dynamic Benchmarking Environment for Android UI Automation Tasks" - Provides detailed methodology and task definitions
• Agent-S: "Agent-S: A Hierarchical Agent Architecture for Complex Task Execution" - Describes the hierarchical planning and execution framework

Community Resources

• Android World Issues: GitHub Issues - For bug reports and feature requests
• Agent-S Discussions: GitHub Discussions - For questions and community support