Skip to content

[DIT-218] 5xx 에러율 디스코드 알림 형식 개선 #274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hyoj-dev merged 1 commit into from
Jun 30, 2026
Merged

[DIT-218] 5xx 에러율 디스코드 알림 형식 개선 #274

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 48 additions & 4 deletions deploy/prod/grafana/provisioning/alerting/contact-points.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,59 @@ contactPoints:
settings:
url: $DISCORD_MONITOR_WEBHOOK_URL
use_discord_username: false
title: '{{ if eq .Status "firing" }}{{ else }}[복구] {{ end }} {{ .CommonLabels.alertname }}'
title: '{{ if eq .CommonLabels.alertname "[CRITICAL] 5xx 에러율 1% 초과" }}{{ if eq .Status "firing" }}{{ .CommonLabels.alertname }}{{ else }}[RESOLVED] 5xx 에러율 정상화{{ end }}{{ else }}{{ if eq .Status "firing" }}{{ else }}[복구] {{ end }}{{ .CommonLabels.alertname }}{{ end }}'
message: |
{{ if eq .CommonLabels.alertname "[CRITICAL] 5xx 에러율 1% 초과" }}
{{ $notificationStatus := .Status }}

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Escape Go-template variables in contact point settings

When Grafana provisions contact point settings, it expands $variable references before storing the template; with no notificationStatus environment variable, this becomes {{ := .Status }} (and the later eq $notificationStatus is also stripped), leaving an invalid notification template. In the normal prod provisioning path this can break Discord notifications for this receiver; use $$notificationStatus or avoid the local variable.

Useful? React with 👍 / 👎.

{{ range .Alerts }}
{{ if eq $notificationStatus "firing" }}
Time : {{ .StartsAt | tz "Asia/Seoul" | date "2006-01-02 15:04:05" }} (KST)
5xx Rate : {{ .Annotations.error_rate }}

────────────────────────

{{ .Annotations.summary }}

확인 필요
{{ .Annotations.description }}


────────────────────────

Explore Logs
{{ .Annotations.explore_url }}

Alert
{{ .GeneratorURL }}

Dashboard
{{ .DashboardURL }}
{{ else }}
Detected : {{ .StartsAt | tz "Asia/Seoul" | date "2006-01-02 15:04:05" }} (KST)
Resolved : {{ .EndsAt | tz "Asia/Seoul" | date "2006-01-02 15:04:05" }} (KST)
Duration : {{ .EndsAt.Sub .StartsAt }}

최근 5분 기준 5xx 에러율이 1% 이하로 내려갔습니다.

────────────────────────

Explore Logs
{{ .Annotations.explore_url }}

Alert
{{ .GeneratorURL }}

Dashboard
{{ .DashboardURL }}
{{ end }}
{{ end }}
{{ else }}
{{ range .Alerts }}
{{ .Annotations.summary }}

확인 필요
{{ .Annotations.description }}

관련 모니터링 링크
- Grafana Alert: {{ .GeneratorURL }}
{{ end }}

{{ end }}
77 changes: 9 additions & 68 deletions deploy/prod/grafana/provisioning/alerting/rules.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
apiVersion: 1

deleteRules:
- orgId: 1
uid: didit-http-5xx-detected

groups:
- orgId: 1
name: didit-prod-critical
Expand Down Expand Up @@ -69,74 +73,6 @@ groups:
- 최근 트래픽 증가 여부
- 메모리 누수 및 OOM 로그

- uid: didit-http-5xx-detected
title: '[CRITICAL] 5xx 에러 발생'
condition: C
data:
- refId: A
datasourceUid: prometheus
relativeTimeRange:
from: 120
to: 0
model:
refId: A
expr: |
sum(
increase(
http_server_requests_seconds_count{
environment="prod",
status=~"5.."
}[1m]
)
)
instant: true
intervalMs: 1000
maxDataPoints: 43200

- refId: B
datasourceUid: __expr__
model:
refId: B
type: reduce
expression: A
reducer: last
datasource:
type: __expr__
uid: __expr__

- refId: C
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [ 0 ]
operator:
type: and
query:
params: [ C ]
reducer:
type: last
type: query
datasource:
type: __expr__
uid: __expr__

noDataState: OK
execErrState: Alerting
for: 0s
labels:
severity: critical
annotations:
summary: 5xx 에러가 발생했습니다.
description: |
- API 서버 ERROR 로그
- 발생 API 및 예외 내용
- DB 및 외부 연동 서비스 상태

- uid: didit-http-5xx-rate-critical
title: '[CRITICAL] 5xx 에러율 1% 초과'
condition: C
Expand Down Expand Up @@ -204,14 +140,19 @@ groups:
noDataState: OK
execErrState: Alerting
for: 2m
dashboardUid: dfp8jkcluca2oe

labels:
severity: critical

annotations:
summary: 5xx 에러율이 1%를 초과했습니다.
error_rate: '{{ humanizePercentage $values.B.Value }}'
description: |
- API 서버 로그
- 최근 배포 여부
- 특정 API 및 외부 서비스 장애 여부
explore_url: https://monitoring.didit.ai.kr/explore?schemaVersion=1&panes=%7B%22ttc%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bcontainer%3D~%5C%22didit-api-%28blue%7Cgreen%29%5C%22%7D%20%7C%3D%20%5C%22ERROR%5C%22%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22code%22%7D%5D,%22range%22:%7B%22from%22:%22now-15m%22,%22to%22:%22now%22%7D,%22panelsState%22:%7B%22logs%22:%7B%22columns%22:%7B%220%22:%22Time%22,%221%22:%22Line%22%7D,%22visualisationType%22:%22table%22,%22labelFieldName%22:%22labels%22%7D%7D%7D%7D&orgId=1

- uid: didit-hikaricp-usage-critical
title: '[CRITICAL] HikariCP 커넥션 풀 사용률 90% 초과'
Expand Down
Loading