Skip to content

feat: Add alerting system for indexer errors#161

Merged
Kevin737866 merged 3 commits into
Kevin737866:mainfrom
gracious01-tech:feat/indexer-error-alerting
Jun 1, 2026
Merged

feat: Add alerting system for indexer errors#161
Kevin737866 merged 3 commits into
Kevin737866:mainfrom
gracious01-tech:feat/indexer-error-alerting

Conversation

@gracious01-tech
Copy link
Copy Markdown
Contributor

Description

Implemented a critical error alerting system for the indexer to improve operational monitoring and incident response.

This update introduces automated alert notifications for critical failures and abnormal indexer behavior using external notification channels such as Slack and email.

The goal is to reduce incident detection time and improve production reliability.


Related Issue

Closes #143


Type of Change

  • Observability improvement
  • Monitoring / alerting enhancement
  • Backend infrastructure update
  • Bug fix
  • New feature
  • Refactor

Changes Included

Alerting Features

  • Added critical error alert pipeline
  • Integrated Slack notification support
  • Added email alert support
  • Configurable alert thresholds
  • Deduplication logic to avoid alert spam
  • Structured alert payloads with context metadata

Monitoring Improvements

  • Alert triggers for:
    • ingestion failures
    • repeated processing errors
    • indexer crashes
    • high error-rate conditions
    • latency threshold violations

Infrastructure Updates

  • Added reusable alert service
  • Added configurable alert providers
  • Improved error logging metadata
  • Added environment-based alert configuration

Problem Solved

Previously:

  • Critical indexer failures could go unnoticed
  • No automated incident notification existed
  • Operators relied on manual monitoring
  • Error escalation was delayed

This PR introduces proactive operational alerting.


Security Impact

No direct security impact.

Improves operational reliability and incident response visibility.


Testing

  • Verified Slack alerts trigger correctly
  • Verified email alerts trigger correctly
  • Tested alert deduplication behavior
  • Simulated ingestion failures and crash scenarios
  • Verified alert payload formatting
  • Existing functionality remains unaffected

Manual Testing Steps

  • Triggered simulated indexer failure
  • Verified Slack notification delivery
  • Verified email notification delivery
  • Triggered repeated identical errors
  • Confirmed deduplication prevents alert spam
  • Tested alert recovery after service restart

Breaking Changes

No breaking changes.

Alerting is configurable and optional through environment settings.


Checklist

  • Code builds successfully
  • Tests pass
  • Follows project conventions
  • No sensitive data exposed
  • Existing indexer functionality preserved

Additional Context

This alerting system improves production readiness by enabling faster detection of failures and improving observability for indexer operations.

The architecture is designed to support additional notification providers in the future.


Reviewer Notes

Please focus on:

  • Alert trigger correctness
  • Deduplication logic reliability
  • Slack/email provider integration
  • Failure handling during notification delivery
  • Configuration flexibility

@drips-wave
Copy link
Copy Markdown

drips-wave Bot commented May 31, 2026

@gracious01-tech Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@Kevin737866 Kevin737866 merged commit 8ba27fc into Kevin737866:main Jun 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add alerting for indexer errors

2 participants