监控与可观测性设计文档

📊 概述

Amazon Pilot 采用 Prometheus + Grafana 的监控方案，实现了完整的 RED (Rate, Errors, Duration) 指标监控体系。本文档详细说明了系统的监控架构、指标设计、以及具体的 PromQL 查询配置。

🏗️ 监控架构

技术栈

Prometheus: 时序数据库，负责指标收集和存储
Grafana: 可视化平台，提供监控仪表板
Node Exporter: 系统级指标收集
Redis Exporter: Redis 指标收集
Custom Metrics: Gateway 自定义业务指标

架构图

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Gateway   │────▶│ Prometheus  │────▶│   Grafana   │
│  (metrics)  │     │   (9090)    │     │   (3001)    │
└─────────────┘     └─────────────┘     └─────────────┘
       │                    ▲                    │
       │                    │                    │
       ▼                    │                    ▼
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│  Services   │     │   Exporters │     │  Dashboard  │
│(auth,product)     │(node,redis) │     │   (RED)     │
└─────────────┘     └─────────────┘     └─────────────┘

📈 Gateway Prometheus 指标

核心指标定义

Gateway 暴露以下 Prometheus 指标（端口 8080/metrics）：

1. HTTP 请求总数

指标名: amazon_pilot_http_requests_total
类型: Counter
标签: service, method, path, status
用途: 统计请求速率（QPS）

2. HTTP 请求耗时

指标名: amazon_pilot_http_request_duration_milliseconds
类型: Histogram
标签: service, method, path
桶定义: [1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]
用途: 统计响应时间分布

3. HTTP 错误总数

指标名: amazon_pilot_http_errors_total
类型: Counter
标签: service, method, path, status
用途: 统计错误率

4. 活跃连接数

指标名: amazon_pilot_active_connections
类型: Gauge
标签: service
用途: 监控并发连接

5. 服务健康状态

指标名: amazon_pilot_service_health
类型: Gauge
标签: service
值: 1=健康, 0=不健康
用途: 服务健康检查

6. JWT 认证统计

指标名: amazon_pilot_jwt_auth_total
类型: Counter
标签: service, result (success/failure)
用途: 监控认证成功率

7. 限流统计

指标名: amazon_pilot_rate_limit_total
类型: Counter
标签: service, plan, result (allowed/blocked)
用途: 监控限流触发情况

🎯 RED 监控 PromQL 查询

Rate (请求速率)

# 整体 QPS
sum(rate(amazon_pilot_http_requests_total[1m]))

# 按服务划分的 QPS
sum by (service) (rate(amazon_pilot_http_requests_total[1m]))

# 按状态码划分的 QPS
sum by (status) (rate(amazon_pilot_http_requests_total[1m]))

# 按 HTTP 方法划分的 QPS
sum by (method) (rate(amazon_pilot_http_requests_total[1m]))

# 特定服务的 QPS (例如 product 服务)
sum(rate(amazon_pilot_http_requests_total{service="product"}[1m]))

# Top 10 最高请求量的 API 路径
topk(10, sum by (path) (rate(amazon_pilot_http_requests_total[5m])))

# 请求增长率（相比 5 分钟前）
rate(amazon_pilot_http_requests_total[1m]) / rate(amazon_pilot_http_requests_total[1m] offset 5m)

Errors (错误率)

# 整体错误率（百分比）
sum(rate(amazon_pilot_http_requests_total{status=~"4..|5.."}[1m]))
/ sum(rate(amazon_pilot_http_requests_total[1m])) * 100

# 按服务划分的错误率
sum by (service) (rate(amazon_pilot_http_requests_total{status=~"4..|5.."}[1m]))
/ sum by (service) (rate(amazon_pilot_http_requests_total[1m])) * 100

# 4xx 客户端错误率
sum(rate(amazon_pilot_http_requests_total{status=~"4.."}[1m]))
/ sum(rate(amazon_pilot_http_requests_total[1m])) * 100

# 5xx 服务端错误率
sum(rate(amazon_pilot_http_requests_total{status=~"5.."}[1m]))
/ sum(rate(amazon_pilot_http_requests_total[1m])) * 100

# 使用专门的错误指标计算错误数
sum(rate(amazon_pilot_http_errors_total[1m]))

# 按服务和状态码分组的错误数
sum by (service, status) (rate(amazon_pilot_http_errors_total[1m]))

# 错误率趋势（5 分钟移动平均）
avg_over_time(
  (sum(rate(amazon_pilot_http_requests_total{status=~"5.."}[1m]))
  / sum(rate(amazon_pilot_http_requests_total[1m])))[5m:]
) * 100

Duration (延迟/耗时)

# P50 延迟（中位数）- 毫秒
histogram_quantile(0.5,
  sum by (service, le) (rate(amazon_pilot_http_request_duration_milliseconds_bucket[5m]))
)

# P90 延迟 - 毫秒
histogram_quantile(0.9,
  sum by (service, le) (rate(amazon_pilot_http_request_duration_milliseconds_bucket[5m]))
)

# P95 延迟 - 毫秒
histogram_quantile(0.95,
  sum by (service, le) (rate(amazon_pilot_http_request_duration_milliseconds_bucket[5m]))
)

# P99 延迟 - 毫秒
histogram_quantile(0.99,
  sum by (service, le) (rate(amazon_pilot_http_request_duration_milliseconds_bucket[5m]))
)

# 平均延迟 - 毫秒
sum by (service) (rate(amazon_pilot_http_request_duration_milliseconds_sum[5m]))
/ sum by (service) (rate(amazon_pilot_http_request_duration_milliseconds_count[5m]))

# 按路径统计的 P95 延迟
histogram_quantile(0.95,
  sum by (path, le) (rate(amazon_pilot_http_request_duration_milliseconds_bucket[5m]))
)

# 慢请求（>1秒）的比例
(sum(rate(amazon_pilot_http_request_duration_milliseconds_bucket{le="+Inf"}[5m]))
- sum(rate(amazon_pilot_http_request_duration_milliseconds_bucket{le="1000"}[5m])))
/ sum(rate(amazon_pilot_http_request_duration_milliseconds_count[5m])) * 100

# 延迟 SLO 达成率（95% 请求 < 500ms）
sum(rate(amazon_pilot_http_request_duration_milliseconds_bucket{le="500"}[5m]))
/ sum(rate(amazon_pilot_http_request_duration_milliseconds_count[5m])) * 100

🔍 业务指标查询

服务健康监控

# 服务健康状态
amazon_pilot_service_health

# 不健康服务数量
count(amazon_pilot_service_health == 0)

# 服务可用率（过去 1 小时）
avg_over_time(amazon_pilot_service_health[1h]) * 100

连接监控

# 活跃连接数
sum by (service) (amazon_pilot_active_connections)

# 总活跃连接数
sum(amazon_pilot_active_connections)

# 连接数峰值（过去 1 小时）
max_over_time(sum(amazon_pilot_active_connections)[1h:])

认证监控

# JWT 认证成功率
sum(rate(amazon_pilot_jwt_auth_total{result="success"}[5m]))
/ sum(rate(amazon_pilot_jwt_auth_total[5m])) * 100

# 认证失败次数
sum(rate(amazon_pilot_jwt_auth_total{result="failure"}[5m]))

# 按服务统计认证情况
sum by (service, result) (rate(amazon_pilot_jwt_auth_total[5m]))

限流监控

# 限流触发次数
sum by (service, result) (rate(amazon_pilot_rate_limit_total[5m]))

# 限流阻塞率
sum(rate(amazon_pilot_rate_limit_total{result="blocked"}[5m]))
/ sum(rate(amazon_pilot_rate_limit_total[5m])) * 100

# 按计划类型统计限流情况
sum by (plan, result) (rate(amazon_pilot_rate_limit_total[5m]))

Redis缓存监控

# Redis 连接数
redis_connected_clients

# Redis 内存使用量（字节）
redis_memory_used_bytes

# Redis 内存使用率（百分比）
(redis_memory_used_bytes / redis_memory_max_bytes) * 100

# Redis 命令处理速率
rate(redis_commands_processed_total[5m])

# Redis 缓存命中率
rate(redis_keyspace_hits_total[5m]) / (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m])) * 100

# Redis 过期键数量
rate(redis_expired_keys_total[5m])

# Redis 驱逐键数量
rate(redis_evicted_keys_total[5m])

# Redis 阻塞客户端数量
redis_blocked_clients

# Redis 键空间统计
redis_db_keys{db="db0"}

# Redis 慢查询监控
redis_slowlog_length

# 按缓存类型统计键数量（需要自定义标签）
redis_db_keys{db="db0", key_pattern="amazon_pilot:product_data:*"}
redis_db_keys{db="db0", key_pattern="amazon_pilot:price:*"}
redis_db_keys{db="db0", key_pattern="amazon_pilot:ranking:*"}

缓存业务指标

# 缓存命中率（应用层统计）
# 需要在应用中添加自定义指标
amazon_pilot_cache_operations_total{result="hit"} / amazon_pilot_cache_operations_total * 100

# 缓存操作响应时间
histogram_quantile(0.95,
  sum by (operation, le) (rate(amazon_pilot_cache_duration_milliseconds_bucket[5m]))
)

# 缓存失效次数
rate(amazon_pilot_cache_invalidations_total[5m])

# 按缓存类型分组的操作统计
sum by (cache_type, result) (rate(amazon_pilot_cache_operations_total[5m]))

📐 Grafana Dashboard 配置

Dashboard 结构

Row 1: 概览指标

当前 QPS (Stat Panel)
- Query: sum(rate(amazon_pilot_http_requests_total[1m]))
- Unit: reqps
- Thresholds: 0-100 (绿), 100-500 (黄), >500 (红)
错误率 (Stat Panel)
- Query: sum(rate(amazon_pilot_http_requests_total{status=~"5.."}[1m])) / sum(rate(amazon_pilot_http_requests_total[1m])) * 100
- Unit: percent
- Thresholds: 0-1 (绿), 1-5 (黄), >5 (红)
P95 延迟 (Stat Panel)
- Query: histogram_quantile(0.95, sum(rate(amazon_pilot_http_request_duration_milliseconds_bucket[5m])))
- Unit: ms
- Thresholds: 0-500 (绿), 500-1000 (黄), >1000 (红)
服务健康 (Stat Panel)
- Query: count(amazon_pilot_service_health == 1) / count(amazon_pilot_service_health) * 100
- Unit: percent
- Thresholds: 100 (绿), 90-99 (黄), <90 (红)

Row 2: RED 指标趋势

请求速率趋势 (Graph Panel)
- Query: sum by (service) (rate(amazon_pilot_http_requests_total[1m]))
- Legend: {{service}}
- Stack: false
错误率趋势 (Graph Panel)
- Query: sum by (service) (rate(amazon_pilot_http_requests_total{status=~"4..|5.."}[1m])) / sum by (service) (rate(amazon_pilot_http_requests_total[1m])) * 100
- Legend: {{service}}
- Alert line: 5%
响应时间分布 (Graph Panel)
- Queries:
  - P50: histogram_quantile(0.5, sum by (le) (rate(amazon_pilot_http_request_duration_milliseconds_bucket[5m])))
  - P90: histogram_quantile(0.9, sum by (le) (rate(amazon_pilot_http_request_duration_milliseconds_bucket[5m])))
  - P95: histogram_quantile(0.95, sum by (le) (rate(amazon_pilot_http_request_duration_milliseconds_bucket[5m])))
  - P99: histogram_quantile(0.99, sum by (le) (rate(amazon_pilot_http_request_duration_milliseconds_bucket[5m])))

Row 3: 服务详情

服务 QPS 分布 (Pie Chart)
- Query: sum by (service) (rate(amazon_pilot_http_requests_total[5m]))
Top 10 API 端点 (Table Panel)
- Query: topk(10, sum by (path) (rate(amazon_pilot_http_requests_total[5m])))
- Columns: Path, QPS
活跃连接数 (Graph Panel)
- Query: sum by (service) (amazon_pilot_active_connections)
JWT 认证成功率 (Gauge Panel)
- Query: sum(rate(amazon_pilot_jwt_auth_total{result="success"}[5m])) / sum(rate(amazon_pilot_jwt_auth_total[5m])) * 100
- Thresholds: >95 (绿), 90-95 (黄), <90 (红)

Row 4: Redis缓存监控

Redis 内存使用率 (Gauge Panel)
- Query: (redis_memory_used_bytes / redis_memory_max_bytes) * 100
- Unit: percent
- Thresholds: 0-70 (绿), 70-85 (黄), >85 (红)
Redis 缓存命中率 (Stat Panel)
- Query: rate(redis_keyspace_hits_total[5m]) / (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m])) * 100
- Unit: percent
- Thresholds: >90 (绿), 70-90 (黄), <70 (红)
Redis 连接数 (Graph Panel)
- Query: redis_connected_clients
- Legend: 连接数
缓存键分布 (Pie Chart)
- Queries:
  - 产品数据: redis_db_keys{db="db0", key_pattern="amazon_pilot:product_data:*"}
  - 价格数据: redis_db_keys{db="db0", key_pattern="amazon_pilot:price:*"}
  - 排名数据: redis_db_keys{db="db0", key_pattern="amazon_pilot:ranking:*"}
Redis 命令速率 (Graph Panel)
- Query: rate(redis_commands_processed_total[5m])
- Legend: Commands/sec
缓存失效趋势 (Graph Panel)
- Queries:
  - 过期键: rate(redis_expired_keys_total[5m])
  - 驱逐键: rate(redis_evicted_keys_total[5m])

🚨 告警规则配置

Prometheus Alert Rules

groups:
  - name: amazon_pilot_gateway
    interval: 30s
    rules:
      # 高错误率告警
      - alert: HighErrorRate
        expr: |
          sum by (service) (rate(amazon_pilot_http_requests_total{status=~"5.."}[5m]))
          / sum by (service) (rate(amazon_pilot_http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
          component: gateway
        annotations:
          summary: "服务 {{ $labels.service }} 错误率过高"
          description: "服务 {{ $labels.service }} 的 5xx 错误率超过 5%，当前值: {{ $value | humanizePercentage }}"
          runbook_url: "https://wiki.example.com/runbooks/high-error-rate"

      # 高延迟告警
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95,
            sum by (service, le) (rate(amazon_pilot_http_request_duration_milliseconds_bucket[5m]))
          ) > 1000
        for: 5m
        labels:
          severity: warning
          component: gateway
        annotations:
          summary: "服务 {{ $labels.service }} 延迟过高"
          description: "服务 {{ $labels.service }} 的 P95 延迟超过 1 秒，当前值: {{ $value }}ms"

      # 服务不健康告警
      - alert: ServiceDown
        expr: amazon_pilot_service_health == 0
        for: 1m
        labels:
          severity: critical
          component: service
        annotations:
          summary: "服务 {{ $labels.service }} 不健康"
          description: "服务 {{ $labels.service }} 健康检查失败超过 1 分钟"

      # QPS 激增告警
      - alert: TrafficSpike
        expr: |
          sum(rate(amazon_pilot_http_requests_total[1m]))
          / sum(rate(amazon_pilot_http_requests_total[1m] offset 5m)) > 2
        for: 2m
        labels:
          severity: warning
          component: gateway
        annotations:
          summary: "流量激增"
          description: "当前 QPS 相比 5 分钟前增长超过 2 倍"

      # 限流触发告警
      - alert: RateLimitTriggered
        expr: |
          sum(rate(amazon_pilot_rate_limit_total{result="blocked"}[5m])) > 10
        for: 5m
        labels:
          severity: warning
          component: gateway
        annotations:
          summary: "限流频繁触发"
          description: "过去 5 分钟限流阻塞请求数超过 10 个/秒"

      # JWT 认证失败率高
      - alert: HighAuthFailureRate
        expr: |
          sum(rate(amazon_pilot_jwt_auth_total{result="failure"}[5m]))
          / sum(rate(amazon_pilot_jwt_auth_total[5m])) > 0.1
        for: 5m
        labels:
          severity: warning
          component: auth
        annotations:
          summary: "JWT 认证失败率过高"
          description: "JWT 认证失败率超过 10%，当前值: {{ $value | humanizePercentage }}"

  - name: amazon_pilot_redis
    interval: 30s
    rules:
      # Redis 内存使用率告警
      - alert: RedisHighMemoryUsage
        expr: (redis_memory_used_bytes / redis_memory_max_bytes) * 100 > 85
        for: 5m
        labels:
          severity: warning
          component: redis
        annotations:
          summary: "Redis 内存使用率过高"
          description: "Redis 内存使用率超过 85%，当前值: {{ $value | humanizePercentage }}"

      # Redis 缓存命中率低告警
      - alert: RedisLowHitRate
        expr: |
          rate(redis_keyspace_hits_total[5m])
          / (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m])) * 100 < 70
        for: 10m
        labels:
          severity: warning
          component: redis
        annotations:
          summary: "Redis 缓存命中率过低"
          description: "Redis 缓存命中率低于 70%，当前值: {{ $value | humanizePercentage }}"

      # Redis 连接数异常
      - alert: RedisHighConnections
        expr: redis_connected_clients > 100
        for: 5m
        labels:
          severity: warning
          component: redis
        annotations:
          summary: "Redis 连接数过高"
          description: "Redis 连接数超过 100，当前值: {{ $value }}"

      # Redis 键驱逐告警
      - alert: RedisKeyEviction
        expr: rate(redis_evicted_keys_total[5m]) > 10
        for: 5m
        labels:
          severity: warning
          component: redis
        annotations:
          summary: "Redis 键驱逐频繁"
          description: "Redis 每秒驱逐键数量超过 10，当前值: {{ $value }}"

      # Redis 不可达告警
      - alert: RedisDown
        expr: up{job="redis-exporter"} == 0
        for: 1m
        labels:
          severity: critical
          component: redis
        annotations:
          summary: "Redis 服务不可达"
          description: "Redis 服务已经不可达超过 1 分钟"

🎯 SLI/SLO 定义

服务级别指标 (SLI)

可用性 SLI

sum(rate(amazon_pilot_http_requests_total{status!~"5.."}[5m]))
/ sum(rate(amazon_pilot_http_requests_total[5m]))

延迟 SLI

sum(rate(amazon_pilot_http_request_duration_milliseconds_bucket{le="500"}[5m]))
/ sum(rate(amazon_pilot_http_request_duration_milliseconds_count[5m]))

错误率 SLI

1 - (sum(rate(amazon_pilot_http_requests_total{status=~"5.."}[5m]))
/ sum(rate(amazon_pilot_http_requests_total[5m])))

服务级别目标 (SLO)

指标	目标	测量窗口
可用性	99.9%	30 天滚动窗口
P95 延迟 < 500ms	95%	7 天滚动窗口
P99 延迟 < 1000ms	99%	7 天滚动窗口
错误率	< 1%	1 天滚动窗口

Error Budget 计算

# 月度错误预算剩余（基于 99.9% SLO）
(1 - 0.999) - (1 - (
  sum(increase(amazon_pilot_http_requests_total{status!~"5.."}[30d]))
  / sum(increase(amazon_pilot_http_requests_total[30d]))
))

📊 性能基准

负载测试结果

场景	QPS	P50 延迟	P95 延迟	P99 延迟	错误率
正常负载	100	15ms	45ms	95ms	0.01%
高负载	500	25ms	85ms	180ms	0.05%
峰值负载	1000	45ms	150ms	450ms	0.5%
压力测试	2000	120ms	850ms	2500ms	5.2%

🔧 调优建议

监控优化

指标采集间隔: 15s (平衡精度与性能)
数据保留策略: 原始数据 15 天，5 分钟聚合 30 天，1 小时聚合 1 年
告警评估间隔: 30s
告警去重: 基于 alertname + service 标签

性能优化

连接池配置: MaxIdleConns=100, MaxOpenConns=200
超时设置: ReadTimeout=30s, WriteTimeout=30s
限流配置: 每服务 100 QPS，突发 200
缓存策略: 热点数据 5 分钟 TTL

📚 参考资料

🚀 快速开始

访问监控系统

# Prometheus
http://localhost:9090

# Grafana (admin/admin123)
http://localhost:3001

# Gateway Metrics Endpoint
http://localhost:8080/metrics

导入 Dashboard

登录 Grafana
导航到 Dashboards → Import
上传 deployments/grafana/dashboards/gateway-red.json
选择 Prometheus 数据源
点击 Import

验证指标

# 检查 Gateway 指标
curl http://localhost:8080/metrics | grep amazon_pilot

# 测试查询
curl -G http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(amazon_pilot_http_requests_total[1m]))'

最后更新: 2024-12-15 版本: v1.0 维护者: Amazon Pilot Team

FilesExpand file tree

MONITORING.md

Latest commit

History