Repository navigation
fix: make EduIDEWarmPoolEmpty detect a warm pool that is gone #38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -324,3 +324,20 @@ monitoring: | |
| volumePercent: 85 | ||
| # -- Days before certificate expiry to start warning. | ||
| certExpiryDays: 21 | ||
| # -- How long an environment's warm pool must be empty before alerting. | ||
| # | ||
| # Shorter than the other component alerts on purpose. An empty warm pool | ||
| # is immediately visible to students - if the instance deployments are | ||
| # gone rather than unhealthy, session launches fail outright - so ten | ||
| # minutes is long enough for a real outage to pass unnoticed. Long enough, | ||
| # though, that recycling the pool during an image bump does not page. | ||
| warmPoolFor: 5m | ||
|
Comment on lines
+329
to
+334
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win Update the duration rationale to match the new default.
🤖 Prompt for AI Agents |
||
| # -- How far back to look for evidence that an environment keeps a warm | ||
| # pool at all. | ||
| # | ||
| # kube-state-metrics emits nothing for a Deployment that does not exist, | ||
| # so a deleted warm pool leaves no series to compare against. Looking back | ||
| # over this window keeps the environment in the expression after its | ||
| # instances vanish. An environment that has genuinely stopped running a | ||
| # warm pool stops alerting once this window passes. | ||
| warmPoolHistory: 6h | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
Repository: EduIDE/EduIDE-Helm
Length of output: 41242
🏁 Script executed:
Repository: EduIDE/EduIDE-Helm
Length of output: 50375
🏁 Script executed:
Repository: EduIDE/EduIDE-Helm
Length of output: 3078
🏁 Script executed:
Repository: EduIDE/EduIDE-Helm
Length of output: 9106
🌐 Web query:
Prometheus alerting rules annotation $labels expression labels rule labels applied overwrite namespace templating semantics💡 Result:
In Prometheus alerting rules, labels and annotations are distinct fields with specific templating and overwriting semantics [1][2]. Templating Semantics: Both labels and annotations support templating using Go's templating system [3]. You can access data from the firing alert instance using specific variables [1][3]: - $labels: Holds the key/value pairs of the alert instance's labels (e.g., {{ $labels.instance }}) [1][2][3]. - $value: Holds the numeric value of the evaluated expression for that alert instance [1][2][3]. - $externalLabels: Accesses globally configured external labels [1][3]. Labels vs. Annotations: - Labels: These are primary attributes of the alert instance. They are used for identity, grouping, and routing within Alertmanager [4]. When you define additional labels in an alerting rule, any existing conflicting labels (labels already present in the time series returned by your expression) will be overwritten by the values provided in the labels clause [1][2]. - Annotations: These are informational, non-identifying fields intended for metadata like descriptions, runbook links, or summaries [1][4]. They do not affect the identity or routing of the alert, but are available to notification templates [1][4]. While they do not "overwrite" identifying labels, they similarly support templating using the same $labels and $value variables [1][3]. In summary, use labels when you need to group or route alerts, and use annotations for informational content [4]. If you specify a label in the rule that matches a label name from your expression, the rule's value will take precedence [1][2].
Citations:
🌐 Web query:
Prometheus AlertingRule eval annotations labels expand labels final label set $labels implementation💡 Result:
In Prometheus, label and annotation values within AlertingRules can be dynamically templated using the Go templating system [1][2]. When an alerting expression triggers, Prometheus evaluates these templates for every firing alert instance [3][4]. Implementation Details The templating process occurs during the evaluation of the alerting rule [3][5]. The Prometheus rules engine uses a template expander to process the strings provided in the labels and annotations fields [3][5]. Key variables provided to the template include [1][2]: $labels: A map containing the key/value pairs of the alert instance (the metric's labels) [1][6]. $externalLabels: A map of globally configured external labels [1][2]. $value: The numerical value of the expression that triggered the alert [1][2]. $externalURL: The configured external URL of the Prometheus instance [1][2]. Technical Workflow 1. Evaluation: The expression defined in the alerting rule is evaluated against the data [3]. 2. Template Expansion: For each resulting time series, Prometheus populates the template data (including the $labels and $value variables) [3][5]. An expander function is called for every label and annotation value to resolve them into their final strings [3][5]. 3. Final Label Set: The resulting labels are used to uniquely identify the alert [3]. If the expansion process results in a duplicate label set for different alerts, Prometheus will flag an error (ErrDuplicateAlertLabelSet), as the final label set must be unique to correctly track the alert's state [3][7]. 4. Storage and Dispatch: These expanded values are then used in the ALERTS time series and when sending notifications to Alertmanager [3][8]. It is important to note that because these templates are executed during every rule iteration, users are advised to keep them lightweight [4]. Additionally, while template expansion changes values for notifications and displays, using variables like $value in labels is generally discouraged as it can generate highly volatile, unique label sets that could lead to an excessive number of alert instances [8].
Citations:
🌐 Web query:
Prometheus rules/alerting.go AlertingRule Eval expand annotations labels lset smpl.Metric source💡 Result:
In the Prometheus source code, specifically within the rules/alerting.go file, the AlertingRule.Eval method is responsible for evaluating alerting rules [1][2]. It processes the results of a PromQL expression and expands templates for labels and annotations [1][3]. The evaluation process generally follows these steps: 1. Query Execution: The method calls the provided query function to execute the alerting rule's PromQL expression at the given timestamp [1][3]. The result is returned as a promql.Vector [1][2]. 2. Processing Vector Elements: The code iterates over each element (smpl, which is a metric sample) in the resulting vector [1][3]. For each sample, it sets up data for template expansion, including the metric's own labels, external labels, external URL, and the sample's value [1][3]. 3. Template Expansion: An expand function is defined within the loop to process strings (like label or annotation values) through a template engine [1][3]. This engine uses the template.NewTemplateExpander, which makes variables like $labels, $externalLabels, $externalURL, and $value available to the template [1][3]. 4. Building Labels and Annotations: - Labels: The code uses a labels.Builder (lb) initialized with the sample's original labels [1][3]. It deletes the metric name and then adds the rule's configured labels, expanding their values if necessary [1][3]. It also sets the special alert name label [1][3]. - Annotations: It uses a labels.ScratchBuilder (sb) to add the rule's annotations, also expanding their values using the same template engine [1][3]. 5. Alert Creation: The resulting labels and annotations are used to calculate a fingerprint (hash), which serves as a unique identifier for that alert instance [3][4]. If this label set is new or corresponds to an existing alert, the AlertingRule updates or creates an Alert object in the rule's active state map [3][4]. The smpl.Metric.Map is used to extract the key-value pairs from the metric sample to be injected into the template data, allowing users to reference specific metric labels dynamically within their alert messages [1][3][5].
Citations:
Pass the environment namespace to
kubectl.$labels.namespacein annotations refers to the expression label, not the rule-addednamespacelabel. Keep the alert text unchanged. Add-n {{ "{{" }} $labels.namespace {{ "}}" }}tokubectl rollout restartso it targets the affected environment instead of the current context namespace.🤖 Prompt for AI Agents
Source: MCP tools