[Feature]: Performance Benchmark Baseline

[AlbumenJ]

[hageshiame]

Hello, after reviewing the repository code, I have drafted a benchmark testing plan, and I would appreciate it if you could take a look at it.

1.Overview

AgentScope-Java is a reactive programming-based agent development framework designed to simplify the process of building Large Language Model (LLM) applications. To ensure the high performance and stability of the framework, we have developed this comprehensive performance benchmark baseline specification.

2. Performance Goals

2.1 Overall Performance Goals

• Ensure core components meet user experience requirements for response times under typical loads
• Guarantee system stability and scalability under high-concurrency scenarios
• Establish quantifiable performance baselines for continuous monitoring and regression detection

2.2 Hardware Environment Baseline

Component	Specification
CPU	Intel Xeon Platinum 8259CL (2.5 GHz, 48 cores)
Memory	64GB RAM
Storage	NVMe SSD
JVM	OpenJDK 17
OS	Ubuntu 20.04 LTS

3. Core Component Performance Baselines

3.1 Agent Component Performance Baseline

Test Scenario	Metric	Target Value	Test Conditions	Priority
Single Agent Call	P50 Latency	< 5ms	MockModel, No Tool Calls	P0
Single Agent Call	P95 Latency	< 15ms	MockModel, No Tool Calls	P0
Single Agent Call	P99 Latency	< 30ms	MockModel, No Tool Calls	P1
Concurrent Agent Calls	10 Concurrent P50 Latency	< 10ms	MockModel, No Tool Calls	P0
Concurrent Agent Calls	10 Concurrent P95 Latency	< 25ms	MockModel, No Tool Calls	P0
Concurrent Agent Calls	100 Concurrent P50 Latency	< 50ms	MockModel, No Tool Calls	P1
Concurrent Agent Calls	100 Concurrent P95 Latency	< 100ms	MockModel, No Tool Calls	P1
Agent Iteration Loop	Single Iteration P50 Latency	< 8ms	MockModel, Simple Reasoning	P0
Agent Iteration Loop	10 Iterations Total Time	< 100ms	MockModel, Simple Reasoning	P0
Agent Memory Usage	Idle State Memory	< 5MB	Initial State	P1
Agent Memory Usage	Running State Memory Growth	< 1MB/100 Calls	Continuous Calls	P1

3.2 Model Component Performance Baseline

Test Scenario	Metric	Target Value	Test Conditions	Priority
Streaming Response	Time to First Byte (TTFB)	< 20ms	MockModel, 100 tokens	P0
Streaming Response	Full Response Time	< 50ms	MockModel, 100 tokens	P0
Batch Calls	10 Concurrent P50 Latency	< 30ms	MockModel	P0
Batch Calls	10 Concurrent P95 Latency	< 80ms	MockModel	P0
Different Length Inputs	100 Tokens Response Time	< 40ms	MockModel	P1
Different Length Inputs	1000 Tokens Response Time	< 200ms	MockModel	P1
Error Handling	API Call Failure Rate	< 0.1%	Simulated Network Exception	P0
Error Handling	Timeout Retry Success Rate	> 99%	Simulated Timeout Scenario	P0

3.3 Tool Component Performance Baseline

Test Scenario	Metric	Target Value	Test Conditions	Priority
Tool Registration	Register 100 Tools Time	< 100ms	Mock Tools	P1
Tool Lookup	Single Lookup P50 Latency	< 1ms	Registered Tools	P0
Tool Lookup	Single Lookup P95 Latency	< 3ms	Registered Tools	P0
Tool Execution	Sync Execution P50 Latency	< 5ms	Mock Tools	P0
Tool Execution	Sync Execution P95 Latency	< 15ms	Mock Tools	P0
Tool Execution	Async Execution P50 Latency	< 3ms	Mock Tools	P0
Tool Execution	Async Execution P95 Latency	< 10ms	Mock Tools	P0
Parallel Tool Calls	5 Concurrent P50 Latency	< 10ms	Mock Tools	P0
Parallel Tool Calls	5 Concurrent P95 Latency	< 25ms	Mock Tools	P0

3.4 Memory Component Performance Baseline

Test Scenario	Metric	Target Value	Test Conditions	Priority
Message Storage	Single Message Addition Time	< 0.5ms	Simple Text Message	P0
Message Retrieval	100 Messages Retrieval Time	< 2ms	InMemoryMemory	P0
Message Retrieval	1000 Messages Retrieval Time	< 10ms	InMemoryMemory	P1
Memory Cleanup	Clean 1000 Messages Time	< 5ms	InMemoryMemory	P1
Long-term Memory	Query P50 Latency	< 10ms	Mock Long-term Memory	P1
Long-term Memory	Query P95 Latency	< 30ms	Mock Long-term Memory	P1

3.5 Pipeline Component Performance Baseline

Test Scenario	Metric	Target Value	Test Conditions	Priority
Sequential Pipeline Execution	Single Pipeline P50 Latency	< 2ms	3 Simple Operations	P0
Sequential Pipeline Execution	Single Pipeline P95 Latency	< 5ms	3 Simple Operations	P0
Parallel Pipeline Execution	5 Concurrent P50 Latency	< 5ms	3 Simple Operations	P0
Parallel Pipeline Execution	5 Concurrent P95 Latency	< 15ms	3 Simple Operations	P0
Complex Pipeline Orchestration	Complex Pipeline Execution Time	< 20ms	10 Operations Combination	P1
Message Passing	MsgHub Delivery Latency	< 1ms	Single Delivery	P1

4. Throughput Performance Baselines

4.1 Request Throughput

Scenario	Metric	Target Value	Test Conditions	Priority
Single Instance QPS	Average QPS	> 1000 req/s	MockModel, No Tool Calls	P0
Single Instance QPS	Peak QPS	> 2000 req/s	MockModel, No Tool Calls	P1
Concurrent Processing	Max Concurrency	> 500 concurrent	MockModel	P0
Resource Utilization	CPU Usage (80% Load)	< 70%	100 QPS	P1
Resource Utilization	Memory Growth Rate	< 5%/hour	Continuous Operation	P1

4.2 Data Throughput

Scenario	Metric	Target Value	Test Conditions	Priority
Token Processing	Processing Rate	> 50000 tokens/s	MockModel	P1
Message Processing	Processing Rate	> 2000 messages/s	Simple Messages	P1
Tool Invocation	Call Frequency	> 1000 calls/s	Mock Tools	P1

5. Resource Consumption Baselines

5.1 Memory Usage

Component	Metric	Target Value	Test Conditions	Priority
Base Memory	Memory Usage After Startup	< 100MB	No Load	P0
Runtime Memory	Steady-state Memory Usage	< 200MB	10 QPS Load	P0
Memory Growth	Continuous 1-hour Growth	< 20MB	10 QPS Load	P0
GC Impact	Full GC Frequency	< 1 time/hour	Normal Load	P1

5.2 CPU Usage

Scenario	Metric	Target Value	Test Conditions	Priority
Idle State	CPU Usage	< 5%	No Request Processing	P1
Light Load	CPU Usage	< 30%	10 QPS	P0
Heavy Load	CPU Usage	< 70%	100 QPS	P0
Thread Pool	Active Threads	< 100 threads	100 Concurrent	P1

5.3 Network I/O

Scenario	Metric	Target Value	Test Conditions	Priority
API Calls	Average Connections	< 50 connections	100 QPS	P1
Data Transfer	Average Bandwidth	< 1 MB/s	Normal Usage	P1

6. Quality Metrics Baselines

6.1 Error Rate

Scenario	Metric	Target Value	Test Conditions	Priority
API Calls	Failure Rate	< 0.1%	Normal Network Environment	P0
Timeout	Timeout Rate	< 0.5%	Normal Network Environment	P0
Retry	Retry Success Rate	> 99%	Simulated Temporary Failures	P0

6.2 Stability

Scenario	Metric	Target Value	Test Conditions	Priority
Long-term Operation	7-day Stability	> 99.9%	Continuous Load	P0
Memory Leak	Memory Growth	< 10MB/24h	Continuous Load	P0
Concurrency Safety	Concurrent Error Rate	< 0.01%	100 Concurrent	P0

7. Testing Methodology

7.1 Test Environment

• Use standard hardware configuration for testing
• Execute in isolated environments to avoid interference from other processes
• Warm-up time of no less than 5 minutes
• Each test is repeated at least 3 times, taking the average

7.2 Testing Tools

• JMH (Java Microbenchmark Harness) for micro-benchmarking
• Micrometer for metrics collection
• Custom test scripts for complex scenario simulation

7.3 Testing Cycle

• Execute complete tests before each major version release
• Weekly regression testing
• Monthly long-term stability testing

8. Performance Regression Detection

8.1 Detection Thresholds

• P50 latency increase exceeding 15% of baseline value
• P95 latency increase exceeding 20% of baseline value
• Average QPS decrease exceeding 10% of baseline value
• Memory usage growth exceeding 25% of baseline value
• Error rate exceeding 1%

[Alexxigang]

I investigated the current build/test layout locally and I'm working on a scoped first phase for this benchmark baseline issue. The plan is to add a dedicated JMH-based benchmark harness for �gentscope-core plus an initial set of core benchmarks (single agent call, sequential pipeline execution, and MsgHub lifecycle/message distribution) so we can establish a repeatable benchmark foundation before filling in the broader matrix.

[Alexxigang]

Opened PR #1143: #1143

This is the scoped phase-1 JMH benchmark harness discussed above for issue #140.