Feat/dashboard v1#7
Merged
Merged
Conversation
…ty alerts - Added a new `/metrics` endpoint for Prometheus to expose service metrics. - Integrated low-quality alert notifications by invoking `send_low_quality_alert` after each evaluation. - Committed evaluation results to the database immediately after processing each log.
- Introduced new configuration options for email notifications, including SMTP server details and recipient information. - Enhanced the evaluator service's ability to send alerts via email, complementing existing notification systems.
- Added a new metrics module to define and collect various Prometheus metrics for the evaluator service. - Included metrics for total evaluations, evaluation duration, batch evaluations, notifications sent, low-quality alerts, and LLM judge requests. - Implemented functions to record metrics for evaluations, batch evaluations, notifications, and scheduler runs, enhancing observability and performance tracking.
- Added an asynchronous function to send email notifications, including HTML formatting for low-quality alerts and evaluation summaries. - Updated Slack and Discord notification functions to accept a notification type parameter for better categorization. - Integrated email notifications into the low-quality alert and batch evaluation summary processes, ensuring alerts are sent via all configured channels. - Improved logging for notification success and failure across all channels.
…lert handling - Added support for sending email notifications via SMTP, including HTML formatting for low-quality alerts. - Updated Slack and Discord notification functions to accept a notification type parameter for better categorization. - Enhanced low-quality alert handling to include email notifications alongside existing Slack and Discord alerts. - Improved logging for notification success and failure, ensuring better observability of alert delivery.
- Added timing and success/error metrics recording for each log evaluation in the batch process. - Integrated metrics functions to track evaluation duration and success rates, improving observability. - Updated scheduler run metrics to log success or error states after batch evaluations. - Enhanced logging for better tracking of evaluation outcomes.
- Introduced a new metrics module to define and collect various Prometheus metrics for the gateway API. - Implemented metrics for HTTP requests, LLM requests, database queries, and logs saved, enhancing observability. - Added a MetricsMiddleware class for automatic collection of HTTP request metrics in FastAPI applications.
- Included the `prometheus-client` library in the gateway API dependencies to support metrics collection and monitoring.
- Included `prometheus-client`, `aiosmtplib`, and `email-validator` libraries in the evaluator service dependencies to enhance metrics collection, email handling, and input validation capabilities.
…tasource - Introduced a new YAML configuration for Grafana dashboards, defining the LLM Quality Observer dashboard provider. - Added a Prometheus datasource configuration for Grafana, setting it as the default datasource for metrics visualization. - Created a Prometheus configuration file with scrape settings for gateway API, evaluator, and Prometheus itself, enhancing monitoring capabilities.
- Introduced Prometheus and Grafana services in the local Docker Compose configuration for enhanced monitoring and visualization. - Configured Prometheus with necessary command options and volume mappings for its configuration file and data storage. - Set up Grafana with environment variables for admin credentials and volume mappings for provisioning and dashboards. - Added new volumes for Prometheus and Grafana data storage.
- Introduced a new JSON configuration file for the LLM Quality Observer dashboard in Grafana. - Configured multiple panels to visualize metrics from Prometheus, including HTTP request rates, evaluation rates, and notification rates. - Enhanced monitoring capabilities with detailed visualizations for LLM requests, evaluation scores, and scheduler runs. - Set refresh interval and added relevant tags for better organization and accessibility.
…ation - Introduced a detailed guide for setting up email notifications in the LLM Quality Observer system, covering prerequisites, configuration, and testing procedures. - Included specific setup instructions for popular SMTP providers like Gmail, Microsoft 365, SendGrid, AWS SES, and Mailgun. - Documented environment variables required for email notifications and provided troubleshooting tips for common issues. - Added a metrics reference section to monitor email notification performance and delivery rates, enhancing observability and support for email-related metrics.
…tings - Added example environment variables for SMTP configuration to facilitate email notifications. - Included placeholders for SMTP host, port, username, password, and recipient emails to guide users in setting up email notifications.
- Introduced a new dashboard overview image to improve the visual representation of the LLM Quality Observer dashboard in Grafana. - This addition aims to provide users with a clearer understanding of the dashboard layout and metrics displayed.
- Introduced comprehensive documentation for the LLM Quality Observer Grafana dashboard in both Korean and English. - The guides cover dashboard access, structure, performance metrics, quality tracking, and troubleshooting steps, enhancing user understanding and usability. - Included detailed PromQL query explanations and usage tips for effective monitoring and evaluation.
- Enhanced the README with a comprehensive overview of the LLM-Quality-Observer project, including its purpose, key features, and architecture. - Updated service components section to reflect the current architecture and added detailed descriptions of each service. - Improved installation and usage instructions, including environment variable configurations and service verification steps. - Added a roadmap section outlining completed features and future plans for the project. - Included security precautions and contributing guidelines to encourage community involvement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.