CP-35089: add release notes for 1.2.9 (#566)

Author: evan-cz

helm/docs/releases/1.2.9.md

@@ -0,0 +1,53 @@
+## [1.2.9](https://github.com/Cloudzero/cloudzero-agent/compare/v1.2.8...v1.2.9) (2025-11-25)
+
+Release 1.2.9 focuses on quality, scalability, configurability, and consistency. It includes significant improvements to the organization and configurability of the Helm chart, Prometheus 3.x support (with 2.x support preserved for now), an HPA for the webhook server, as well as some early previews of experimental functionality we hope to stabilize over the next few releases.
+
+### Key Improvements
+
+- **Prometheus 3.x Support**: Upgraded default Prometheus version from 2.55.1 to 3.7.3 with automatic backward compatibility. The chart detects Prometheus version and uses the appropriate agent mode flag (`--agent` for 3.x, `--enable-feature=agent` for 2.x). Customers using custom Prometheus 2.x images will continue to work without changes.
+
+- **Webhook Server Autoscaling**: Added Horizontal Pod Autoscaler support for the webhook server, enabling automatic scaling based on CPU and memory utilization. Enable with `insightsController.autoscaling.enabled: true`.
+
+### Configuration Improvements
+
+- **Unified Label/Annotation System**: Refactored label and annotation generation with new `generateLabels` and `generateAnnotations` helpers. Labels now follow Kubernetes recommended practices with `app.kubernetes.io/name` for component identity and `app.kubernetes.io/part-of: cloudzero-agent` for chart membership.
+
+- **Component-Specific Metadata**: Added support for component-specific `labels`, `annotations`, `podLabels`, and `podAnnotations` across all workload types, providing fine-grained control over Kubernetes metadata.
+
+- **Centralized Resource Names**: Implemented unified resource naming pattern (`{release-name}-cz-{component}`) across all Kubernetes resources, improving consistency and enabling programmatic name reconstruction.
+
+- **Unified Mode Configuration**: New `components.agent.mode` property consolidates deployment mode selection with values: `agent`, `server`, `federated`, and `clustered`. Legacy properties continue to work with automatic derivation.
+
+- **Cohesive Replicas System**: New `defaults.replicas` property provides global default with mode-specific constraints.
+
+- **Persistent Volume Strategy**: Changed deployment strategy to `Recreate` when persistent volumes are enabled, preventing volume mount conflicts during rolling updates.
+
+### Reliability Improvements
+
+- **Subchart Isolation**: Excluded `.global` sections from configuration checksum calculation, preventing unnecessary pod restarts when parent chart globals change in subchart deployments.
+
+- **Cert-Manager Compatibility**: Fixed ArgoCD reconciliation failures by removing empty `caBundle` key from webhook configuration when using cert-manager (which injects the CA bundle via annotation).
+
+- **DNS Resolution**: Updated cAdvisor configuration to use fully qualified domain name `kubernetes.default.svc.cluster.local:443` for improved DNS resolution reliability.
+
+### Support Tooling
+
+- **Diagnostic Script**: Added `scripts/anaximander.sh` for comprehensive diagnostic information gathering. Customers can run this script to collect logs, configurations, resource status, and environment context for CloudZero support.
+
+- **Post-Install Guidance**: Updated Helm NOTES.txt with improved post-installation guidance and next steps.
+
+### Experimental Features
+
+The following features are experimental and may change in future releases:
+
+- **Grafana Alloy Integration**: Added Grafana Alloy as an alternative to Prometheus for metrics collection in high-volume environments. Configure with `components.agent.mode: clustered` to enable.
+
+- **GPU Metrics Collection**: Added NVIDIA DCGM GPU metrics scraping. Enable with `prometheusConfig.scrapeJobs.gpu.enabled: true`. Note that this is just for collection, CloudZero does not yet support cost allocation based on GPU.
+
+### Upgrade Steps
+
+This release includes changes to immutable Kubernetes selectors, requiring the `--force` flag to recreate affected resources:
+
+```sh
+helm upgrade --install <RELEASE_NAME> cloudzero/cloudzero-agent -n <NAMESPACE> --create-namespace -f configuration.example.yaml --version 1.2.9 --force
+```