From 0be0c52ee10489ef140e78945249a88e1494c061 Mon Sep 17 00:00:00 2001 From: Rendre Greyling Date: Mon, 31 Aug 2026 14:45:25 +0200 Subject: [PATCH 1/5] First iteration of perf docs Signed-off-by: Rendre Greyling --- .../content/en/docs/12_contributing/_index.md | 2 + .../code-contribution/_index.md | 5 + .../development-environment.md | 4 + .../code-contribution/performance-tests.md | 302 ++++++++++++++++ .../6_configuration_and_deployments/_index.md | 1 + .../configurations/_index.md | 7 +- .../configurations/components/_index.md | 6 +- .../function-runner-config/_index.md | 4 + .../porch-controllers-config/_index.md | 5 + .../components/porch-server-config/_index.md | 4 +- .../configurations/opentelemetry.md | 335 ++++++++++++++++-- .../deployments/_index.md | 11 + .../deployments/catalog-deployment.md | 2 +- .../deployments/local-dev-env-deployment.md | 2 + ...local-performance-monitoring-deployment.md | 125 +++++++ 15 files changed, 784 insertions(+), 31 deletions(-) create mode 100644 docs/content/en/docs/12_contributing/code-contribution/performance-tests.md create mode 100644 docs/content/en/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment.md diff --git a/docs/content/en/docs/12_contributing/_index.md b/docs/content/en/docs/12_contributing/_index.md index 4cbb321d6..004dc9675 100644 --- a/docs/content/en/docs/12_contributing/_index.md +++ b/docs/content/en/docs/12_contributing/_index.md @@ -11,6 +11,8 @@ Thank you for your interest in contributing to Porch! - **[Documentation]({{% relref "docs-contribution" %}})** - Fix errors, clarify concepts, add examples, write guides - **[Code]({{% relref "code-contribution" %}})** - Fix bugs, add features, improve performance, write tests + - [Development Environment]({{% relref "code-contribution/development-environment" %}}) - Kind cluster and debugging + - [Performance Tests]({{% relref "code-contribution/performance-tests" %}}) - Load and soak tests - **Issues** - Report bugs or request features in [GitHub Issues](https://github.com/kptdev/porch/issues) - **Community** - Answer questions and participate in discussions diff --git a/docs/content/en/docs/12_contributing/code-contribution/_index.md b/docs/content/en/docs/12_contributing/code-contribution/_index.md index 3da97b9e1..a0f86c666 100644 --- a/docs/content/en/docs/12_contributing/code-contribution/_index.md +++ b/docs/content/en/docs/12_contributing/code-contribution/_index.md @@ -7,6 +7,11 @@ description: How to contribute to the Porch codebase This guide explains how to set up your Git environment and contribute code to Porch. +Related guides: + +- [Development Environment]({{% relref "development-environment" %}}) - Kind cluster, make targets, and debugging +- [Performance Tests]({{% relref "performance-tests" %}}) - Load and soak tests for package revision lifecycle latency + ## Fork the Repository Porch uses a fork-based workflow. This protects the upstream repository from accidental changes and ensures all modifications go through the pull request review process. diff --git a/docs/content/en/docs/12_contributing/code-contribution/development-environment.md b/docs/content/en/docs/12_contributing/code-contribution/development-environment.md index 5e2c02350..d2df057b5 100644 --- a/docs/content/en/docs/12_contributing/code-contribution/development-environment.md +++ b/docs/content/en/docs/12_contributing/code-contribution/development-environment.md @@ -306,6 +306,10 @@ make test-unit make test-integration ``` +### Performance Tests + +Load and soak tests for package revision lifecycle latency live in `test/performance/`. See [Performance Tests]({{% relref "performance-tests" %}}) for make targets, flags, output files, and metrics. Deploy [local performance monitoring]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}) before running tests with `-enable-prometheus=true`. + ### Single Test Case Run specific tests from the command line: diff --git a/docs/content/en/docs/12_contributing/code-contribution/performance-tests.md b/docs/content/en/docs/12_contributing/code-contribution/performance-tests.md new file mode 100644 index 000000000..a04104f72 --- /dev/null +++ b/docs/content/en/docs/12_contributing/code-contribution/performance-tests.md @@ -0,0 +1,302 @@ +--- +title: "Performance Tests" +type: docs +weight: 2 +description: Run Porch package revision lifecycle performance and load tests +--- + +Performance tests measure Porch package revision lifecycle latency and throughput under configurable load. They create Gitea and Porch repositories, run package revisions through the full lifecycle (create → update resources → propose → approve/publish), optionally delete them, and record per-operation timings plus optional Prometheus metrics. + +For the local Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy stack used with these tests, see [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). Server-side metrics and pprof configuration are documented in [OpenTelemetry Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}). + +## Prerequisites + +- Docker +- Kubernetes CLI (`kubectl`) +- [kpt CLI](https://kpt.dev/installation/) +- Go development environment +- A running Porch deployment with Gitea (see [Set Up the Development Environment](#set-up-the-development-environment)) + +## Set Up the Development Environment + +From the repository root, set up the Kind cluster, Gitea, and test repository: + +```bash +make setup-dev-env +``` + +This matches the [Local Development Environment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-dev-env-deployment" %}}) Kind setup. + +## Build and Deploy Porch + +Build and deploy Porch into a Kind cluster. Choose the target that matches the API version and cache backend you want to test: + +| Make target | Cache | v1alpha2 | Notes | +|-------------|-------|----------|-------| +| `make run-in-kind` | CR | no | Default; uses v1alpha1 PackageRevision API | +| `make run-in-kind-db-cache` | DB (PostgreSQL) | no | Database-backed cache | +| `make run-in-kind-v1alpha2` | DB | yes | Required for `-api-version=v1alpha2` tests | +| `make run-in-kind-v1alpha2-no-controller` | DB | yes | Exposes function-runner; run the controller locally | + +```bash +# v1alpha1 (default) +make run-in-kind + +# v1alpha2 with DB cache +make run-in-kind-v1alpha2 +``` + +Verify pods are running in `porch-system` before starting tests. + +To tear down: + +```bash +make destroy +``` + +## Deploy Monitoring (Optional) + +Deploy the monitoring stack before running tests with `-enable-prometheus=true`. See [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}) for make targets, UI URLs, and Grafana dashboards. + +The test process exposes OpenTelemetry metrics on **port 9095** on the host. Prometheus inside Kind scrapes them via the Docker gateway IP (`172.17.0.1:9095`). + +{{% alert title="Warning" color="warning" %}} +Only enable `-enable-prometheus=true` when the monitoring stack is deployed. The test waits 15 seconds before shutting down its metrics server so Prometheus can scrape final values. +{{% /alert %}} + +## Test Packages + +Three packages are available under `test/performance/packages/`. Select one with `-package-path`: + +| Package | Path | Description | +|---------|------|-------------| +| **Small** (default) | `packages/small-package` | Single Deployment; `set-namespace` + `apply-setters` | +| **Complex** | `packages/complex-package` | 10 KRM resource files; 15 kpt catalog functions (mutators + validators) | +| **Large** | `packages/large-package` | 30 microservice Deployments in a single ~550 KB manifest; 15 kpt catalog functions | + +### Large Package Requirements + +The large package generates significantly larger request payloads than the default 6 MB `--max-request-body-size` (set on both porch-server and function-runner). **Only use `packages/large-package` when:** + +1. **Porch server, function-runner, and porch-controllers have adequate CPU and memory requests/limits** for rendering and reconciling large packages with a full kpt pipeline (15 functions across 30 Deployments). +2. **`--max-request-body-size` is increased significantly** on both porch-server and function-runner (they must stay in sync). The default is `6291456` (6 MB) in `deployments/porch/3-porch-server.yaml` and `deployments/porch/2-function-runner.yaml`. + +Default Kind deployment resource profiles: + +| Component | CPU request | CPU limit | Memory request | Memory limit | +|-----------|-------------|-----------|----------------|--------------| +| porch-server | 250m | — | 256Mi | 512Mi | +| function-runner | 125m | — | 64Mi | — | +| porch-controllers | 500m | 1000m | 512Mi | 1Gi | + +These defaults are suitable for small and complex packages. Large-package workloads typically need higher limits across all three components. + +## Run Performance Tests + +Tests are gated by environment variables and skipped otherwise. Run from the repository root or from `test/performance/`: + +```bash +cd test/performance/ +``` + +### Scale / Load Test (`TestPorchScalePerformance`) + +Creates a configured number of repositories, packages, and revisions to simulate load. Runs repositories and packages in parallel up to the configured parallelism limits. + +```bash +LOAD_TEST=1 go test -v ./... -timeout 1h +``` + +Example with custom parameters: + +```bash +LOAD_TEST=1 go test -v ./... \ + -namespace=porch-metrics \ + -api-version=v1alpha1 \ + -repos=2 \ + -packages=3 \ + -revisions=5 \ + -repo-parallelism=2 \ + -package-parallelism=2 \ + -package-path=packages/complex-package \ + -enable-prometheus=true \ + -enable-deletion=true \ + -timeout 2h +``` + +### Maximum Package Revisions Test (`TestIncreasePRsPerformance`) + +Creates package revisions sequentially in a single repository until the error-rate threshold is exceeded. Designed for long-running soak tests. + +```bash +MAX_PR_TEST=1 go test -v ./... -timeout 72h +``` + +This test uses `-error-rate` (default `0.1`, i.e. 0.1%) to decide when to stop: once the fraction of failed revisions reaches the threshold, the test ends. Use a long timeout (72h or more is recommended). + +### API Versions + +| Version | Deploy target | Lifecycle driver | +|---------|---------------|------------------| +| `v1alpha1` (default) | `make run-in-kind` or `make run-in-kind-db-cache` | Direct lifecycle transitions via API | +| `v1alpha2` | `make run-in-kind-v1alpha2` | Controller-reconciled lifecycle with wait steps | + +v1alpha2 additionally records **Wait Ready**, **Wait Rendered**, and **Wait Published** operation timings while the controller reconciles each revision. + +Select the API version with `-api-version=v1alpha1` or `-api-version=v1alpha2`. + +### What Each Test Does + +For every repository (`{namespace}-test-{N}`): + +1. Create a Gitea repository +2. Create a Porch `Repository` CR and wait for Ready +3. For each package (`network-function-{N}`) and revision: + - List existing package revisions + - Create a new package revision (init on v1, copy-from-published on subsequent revisions) + - Update package revision resources from the selected test package + - Propose → approve/publish (v1alpha2 waits for controller reconciliation between steps) +4. Optionally delete all created package revisions (`-enable-deletion=true`) + +Tests handle `SIGINT`/`SIGTERM` gracefully: in-flight work stops and results collected so far are written. + +### Test Parameters + +| Flag | Default | Description | +|------|---------|-------------| +| `-namespace` | `porch-metrics` | Kubernetes namespace for test resources | +| `-api-version` | `v1alpha1` | Porch API version (`v1alpha1` or `v1alpha2`) | +| `-repos` | `1` | Number of repositories | +| `-packages` | `1` | Packages per repository | +| `-revisions` | `1` | Revisions per package (ignored by MAX_PR_TEST, which runs until error threshold) | +| `-repo-parallelism` | `1` | Repositories created in parallel | +| `-package-parallelism` | `1` | Packages created in parallel per repository | +| `-package-path` | `packages/small-package` | Path to package resources directory | +| `-error-rate` | `0.1` | Max failure percentage before MAX_PR_TEST stops (0.1 = 0.1%) | +| `-enable-deletion` | `false` | Delete all package revisions after lifecycle test | +| `-enable-prometheus` | `false` | Expose OTel metrics on host port 9095 | +| `-metrics-log-prefix` | `porch-metrics` | Prefix for timestamped log in `logs/` | +| `-results-file` | `load_test_results.txt` | Approved/deleted revision summary | +| `-detailed-log-file` | `load_test.log` | Per-operation detailed log | +| `-repo-results-csv` | `load_test_lifecycle_results.csv` | Per-revision lifecycle duration CSV | +| `-operations-csv` | `load_test_operations_results.csv` | Per-operation timing CSV | +| `-deletion-csv` | `load_test_deletion_results.csv` | Deletion operation CSV | +| `-gitea-url` | `http://localhost:3000` | Gitea API base URL | +| `-gitea-username` | `porch` | Gitea username | +| `-gitea-password` | `secret` | Gitea password | + +The KRM function registry URL is configured via `PORCH_GHCR_PREFIX_URL` in the repo root `.env` file. It is applied at deploy time to porch-server, function-runner, and porch-controllers (`make run-in-kind`, `make run-in-kind-db-cache`, and `make run-in-kind-v1alpha2` all read `.env` automatically via `make deployment-config`). Package `Kptfile` images use short names (for example `set-namespace:v0.4.1`); porch-server and function-runner resolve them with `--default-image-prefix`, and controllers use the `DEFAULT_IMAGE_PREFIX` environment variable. The `CHANGE_NAMESPACE` placeholder in Kptfiles is substituted at test runtime. + +## Output Files + +Results are written relative to the working directory (typically `test/performance/`): + +| Output | Location | Contents | +|--------|----------|----------| +| Timestamped test log | `logs/{prefix}-{api-version}-{timestamp}.log` | Consolidated results table | +| Results summary | `load_test_results.txt` (configurable) | Approved and deleted revision timings | +| Detailed log | `load_test.log` (configurable) | Per-operation log with configuration header | +| Lifecycle CSV | `load_test_lifecycle_results.csv` | Per package-revision total lifecycle duration | +| Operations CSV | `load_test_operations_results.csv` | Per-operation min/max/avg/total | +| Deletion CSV | `load_test_deletion_results.csv` | Deletion timings (when `-enable-deletion=true`) | + +## Sample Output + +```bash +LOAD_TEST=1 go test -v ./... \ + -namespace=porch-metrics \ + -repos=1 -packages=1 -revisions=3 \ + -enable-prometheus=true \ + -enable-deletion=true \ + -timeout 1h +``` + +``` +=== Consolidated Performance Test Results (v1alpha1) === +Operation Min Max Avg Total +------------------------------------------------------------------------------------ +Create Gitea Repository R0 272ms 272ms 272ms 272ms +Create Porch Repository R0 3ms 3ms 3ms 3ms +Repository Ready Wait R0 2.004s 2.004s 2.004s 2.004s +Package Revision List v1 9ms 9ms 9ms 9ms +Package Revision Create v1 10ms 10ms 10ms 10ms +Package Revision Get Resources v1 3ms 3ms 3ms 3ms +Package Revision Update v1 10ms 10ms 10ms 10ms +Package Revision Get v1 2ms 2ms 2ms 2ms +Package Revision Propose v1 11ms 11ms 11ms 11ms +Package Revision Get (Proposed) v1 2ms 2ms 2ms 2ms +Package Revision Approve/Publish v1 348ms 348ms 348ms 348ms +Package Revision Propose Deletion v1 8ms 8ms 8ms 8ms +Package Revision Delete v1 262ms 262ms 262ms 262ms +... +Total lifecycle duration for all operations: 12.5s +Tests completed! +``` + +With `-api-version=v1alpha2`, additional rows appear for **Wait Ready**, **Wait Rendered**, and **Wait Published** operations. + +## Performance Test Metrics + +When `-enable-prometheus=true`, the test process exports the following OpenTelemetry metrics on `localhost:9095/metrics`. They appear on the **Porch Performance Test Dashboard** in Grafana. + +### Instruments + +| Metric name | Type | Unit | Description | +|-------------|------|------|-------------| +| `porch_perf_operation_duration_seconds` | Histogram | seconds | Duration of each performance-test operation | +| `porch_perf_operations_total` | Counter | — | Total number of performance-test operations | +| `porch_perf_repositories_created_total` | Counter | — | Repositories created during the test | +| `porch_perf_packages_created_total` | Counter | — | Packages created during the test | +| `porch_perf_package_revisions_total` | Counter | — | Package revisions created during the test | +| `porch_perf_lifecycle_transition_duration_seconds` | Histogram | seconds | Duration of package lifecycle transitions | +| `porch_perf_test_run_info` | Gauge | — | Current test-run metadata (value is always `1`) | +| `porch_perf_active_operations` | UpDownCounter | — | Number of currently in-flight operations | + +### Attributes + +| Metric | Attributes | +|--------|------------| +| `porch_perf_operation_duration_seconds`, `porch_perf_operations_total` | `operation`, `api_version`, `repository`, `package`, `status` (`success` or `error`) | +| `porch_perf_package_revisions_total` | `operation`, `status` | +| `porch_perf_lifecycle_transition_duration_seconds` | `from_state`, `to_state`, `api_version`, `repository`, `package`, `status` | +| `porch_perf_test_run_info` | `test_name`, `namespace`, `api_version`, `start_time` | +| `porch_perf_active_operations` | `operation`, `api_version` | +| `porch_perf_repositories_created_total`, `porch_perf_packages_created_total` | none | + +### Operation Labels + +The `operation` attribute uses these keys: + +| Key | When recorded | +|-----|----------------| +| `GITEA-REPO-CREATE` | Create Gitea repository | +| `PORCH-REPO-CREATE` | Create Porch `Repository` CR | +| `REPO-WAIT` | Wait for repository Ready | +| `LIST` | List package revisions | +| `CREATE` | Create package revision | +| `GET-RESOURCES` | Get package revision resources | +| `UPDATE` | Update package revision resources | +| `GET` | Get package revision | +| `PROPOSE` | Propose package revision | +| `GET-PROPOSED` | Get package revision after propose | +| `APPROVE` | Approve / publish package revision | +| `PROPOSE-DELETION` | Propose deletion (when `-enable-deletion=true`) | +| `DELETE` | Delete package revision (when `-enable-deletion=true`) | +| `WAIT-READY` | v1alpha2: wait until the controller marks the revision Ready | +| `WAIT-RENDERED` | v1alpha2: wait until the controller finishes rendering | +| `WAIT-PUBLISHED` | v1alpha2: wait until the controller publishes the revision | + +Histogram bucket boundaries: + +- `porch_perf_operation_duration_seconds`: 0.01, 0.05, 0.1, 0.5, 1, 2, 5, 10, 30, 60, 120 seconds +- `porch_perf_lifecycle_transition_duration_seconds`: 0.1, 0.5, 1, 2, 5, 10, 30, 60 seconds + +### Server-Side Metrics During a Test Run + +While tests run, Prometheus also scrapes Porch components on port 9464. Those series (`porch_api_call_duration_seconds`, `porch_api_requests_by_user`, `porch_package_size_bytes`, and `porch_package_size_bytes_total`) are described in [Available Metrics]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry#available-metrics" %}}). Use the **Porch API** and **Porch File-system Resources** Grafana dashboards alongside the performance-test dashboard. + +## See Also + +- [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}) — Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy +- [OpenTelemetry Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}) — exporters, pprof, and server metrics +- [Development Environment]({{% relref "development-environment" %}}) — local Kind setup and debugging diff --git a/docs/content/en/docs/6_configuration_and_deployments/_index.md b/docs/content/en/docs/6_configuration_and_deployments/_index.md index 3ab90a1f2..30b0ef33a 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/_index.md +++ b/docs/content/en/docs/6_configuration_and_deployments/_index.md @@ -12,6 +12,7 @@ This section covers deploying Porch in various environments and configuring its Start by selecting the appropriate deployment method for your environment: - **[Local Development]({{% relref "deployments/local-dev-env-deployment" %}})** - Kind-based setup for development and testing +- **[Local Performance Monitoring]({{% relref "deployments/local-performance-monitoring-deployment" %}})** - Prometheus, Grafana, Grafana Alloy, Pyroscope and Jaeger for kind local clusters - **[Catalog Deployment]({{% relref "deployments/catalog-deployment" %}})** - Production deployment with pre/post-deployment configuration planning ## 2. Configure Components (Optional) diff --git a/docs/content/en/docs/6_configuration_and_deployments/configurations/_index.md b/docs/content/en/docs/6_configuration_and_deployments/configurations/_index.md index 77ba7b2d1..65d666f67 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/configurations/_index.md +++ b/docs/content/en/docs/6_configuration_and_deployments/configurations/_index.md @@ -18,9 +18,11 @@ Configure individual Porch components: - [Function Runner]({{% relref "components/function-runner-config" %}}) - Function execution environment - [Private Registries]({{% relref "components/function-runner-config/private-registries-config" %}}) - Container registry authentication -### OTEL Metrics & Tracing +### OTEL Metrics, Tracing, and Profiling -[OpenTelemetry]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}) - Tracing and metrics configuration +[OpenTelemetry]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}) - Tracing, metrics, and pprof configuration + +For a local Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy stack, see [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). ## Core Configuration Options @@ -39,5 +41,6 @@ Configure Git repository synchronization with ConfigSync or other GitOps tools. - Start with default CR cache for simplicity - Configure private registries only if using private KRM functions in Function Runner - Enable tracing in development environments for debugging +- Deploy [local performance monitoring]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}) when measuring latency or running [performance tests]({{% relref "/docs/12_contributing/code-contribution/performance-tests" %}}) - Use cert-manager for production TLS certificate management - Set appropriate resource limits for each component diff --git a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/_index.md b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/_index.md index aa849bb48..5d0e3f69e 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/_index.md +++ b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/_index.md @@ -21,4 +21,8 @@ Manage the lifecycle of Repositories, PackageRevisions, PackageVariants, and Pac ### [Function Runner]({{% relref "function-runner-config" %}}) Executes KRM functions in isolated containers: -- [Private Registry Access]({{% relref "function-runner-config/private-registries-config" %}}) - Container registry authentication \ No newline at end of file +- [Private Registry Access]({{% relref "function-runner-config/private-registries-config" %}}) - Container registry authentication + +## Observability + +[OpenTelemetry]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}) covers metrics, traces, and pprof for every component. Deploy the local stack (including Pyroscope and Grafana Alloy) with [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). \ No newline at end of file diff --git a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/function-runner-config/_index.md b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/function-runner-config/_index.md index 89a7c7c0d..e09f03890 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/function-runner-config/_index.md +++ b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/function-runner-config/_index.md @@ -194,6 +194,10 @@ spec: periodSeconds: 5 ``` +## Observability + +For tracing, metrics, and pprof configuration, see [OpenTelemetry Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}). For a local Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy stack, see [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). + {{% alert title="Note" color="primary" %}} For advanced configuration options: - [Pod Templates]({{% relref "pod-templates" %}}) - Customize function pod specifications diff --git a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-controllers-config/_index.md b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-controllers-config/_index.md index 1ee9dd6a4..1bd19a549 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-controllers-config/_index.md +++ b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-controllers-config/_index.md @@ -126,3 +126,8 @@ Both the PackageRevision and Repository webhooks run in the porch-controllers po For webhook TLS certificate setup and management, see [Webhook Certificate Management](../porch-webhooks/cert-manager-webhooks.md). For details on webhook validation rules, see [Webhook Validation Rules](../porch-webhooks/validation-rules.md). + + +## Observability + +For tracing, metrics, and pprof configuration, see [OpenTelemetry Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}). For a local Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy stack, see [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). \ No newline at end of file diff --git a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-server-config/_index.md b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-server-config/_index.md index 7e85e6127..887b12fc2 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-server-config/_index.md +++ b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-server-config/_index.md @@ -70,9 +70,9 @@ env: For detailed Git repository authentication configuration, see [Git Authentication]({{% relref "git-authentication" %}}) subsection. -## Distributed Tracing +## Distributed Tracing, Metrics, and Profiling -For tracing and metrics configuration, see [OpenTelemetry Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}). +For tracing, metrics, and pprof configuration, see [OpenTelemetry Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}). For a local Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy stack, see [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). ## Resource Limits diff --git a/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md b/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md index 5e6a32b7d..1000c8ec0 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md +++ b/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md @@ -2,7 +2,7 @@ title: "OpenTelemetry Configuration" type: docs weight: 4 -description: Configure OpenTelemetry metrics and traces export for Porch components +description: Configure OpenTelemetry metrics, traces, and pprof profiling for Porch components --- ## Overview @@ -11,10 +11,16 @@ Porch supports OpenTelemetry observability through the [autoexport package](http All Porch components (porch-server, porch-controllers, function-runner, and wrapper-server) support OpenTelemetry configuration through standardized environment variables as defined by the [OpenTelemetry specification](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/). +Porch also exposes Go [pprof](https://pkg.go.dev/net/http/pprof) endpoints for continuous profiling. Grafana Alloy can scrape those endpoints into Pyroscope when the pod annotations and `PORCH_PPROF_PORT` environment variable described in [Pprof Configuration](#pprof-configuration) are set. + +Default Kind deployments already export Prometheus metrics on port 9464 (`OTEL_METRICS_EXPORTER=prometheus`) and declare a container port named `pprof`. Trace export and the pprof HTTP server stay disabled until you set the corresponding environment variables. + {{% alert title="Note" color="primary" %}} **Current Implementation Status**: Porch currently implements metrics and traces export. Logs export is not supported. {{% /alert %}} +For a local Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy stack, see [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). For load-test metrics emitted by the test process, see [Performance Tests]({{% relref "/docs/12_contributing/code-contribution/performance-tests" %}}). + ## Traces Configuration ### Quick Start with Jaeger @@ -111,6 +117,8 @@ kubectl port-forward -n porch-system service/jaeger-http 16686 Open http://localhost:16686 and you should see `porch-server`, `porch-function-runner`, and `porch-controllers` in the service dropdown. +To deploy Jaeger as part of the local monitoring stack (namespace `porch-monitoring`) instead of applying the manifest above, see [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). + ### OTLP Trace Export Export traces to an OpenTelemetry Protocol (OTLP) collector using either HTTP or gRPC protocols. @@ -161,6 +169,148 @@ All environment variables apply to all Porch components: porch-server, porch-con | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | Traces-specific endpoint (overrides general endpoint) | - | `http://localhost:4318/v1/traces` | | `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` | Traces-specific protocol (overrides general protocol) | - | `http/protobuf`, `grpc` | +## Pprof Configuration + +Porch components start a pprof HTTP server when `PORCH_PPROF_PORT` is set. Grafana Alloy uses `profiles.grafana.com/*` pod annotations to discover those endpoints and ship profiles to Pyroscope. + +{{% alert title="Warning" color="warning" %}} +Enabling pprof turns on mutex and block profiling (`runtime.SetMutexProfileFraction(1)` and `runtime.SetBlockProfileRate(1)`). That adds overhead. Use it in development and performance-test clusters, not as a default in production, unless you have measured the cost. +{{% /alert %}} + +Pprof applies to porch-server, function-runner, and porch-controllers. wrapper-server does not expose a pprof server. + +### Environment Variable + +| Variable | Description | Default | +|----------|-------------|---------| +| `PORCH_PPROF_PORT` | TCP port for the pprof HTTP server. If unset, the server does not start. | unset (disabled) | + +Set it to match the named container port (typically `8080`): + +```yaml +env: + - name: PORCH_PPROF_PORT + value: "8080" +``` + +### Container Port + +The container must expose a port named `pprof` so Alloy can resolve `profiles.grafana.com/*/port_name: pprof`. Default Kind manifests already declare this port in `deployments/porch/3-porch-server.yaml`, `deployments/porch/2-function-runner.yaml`, and `deployments/porch/9-controllers.yaml`: + +```yaml +ports: + - containerPort: 9464 + name: metrics + protocol: TCP + - containerPort: 8080 + name: pprof + protocol: TCP +``` + +### Pod Annotations for Pyroscope + +Add these annotations on the pod template so Grafana Alloy scrapes the pprof port: + +```yaml +metadata: + annotations: + profiles.grafana.com/service_name: "porch-server" # or porch-function-runner / porch-controllers + profiles.grafana.com/cpu.scrape: "true" + profiles.grafana.com/cpu.port_name: "pprof" + profiles.grafana.com/memory.scrape: "true" + profiles.grafana.com/memory.port_name: "pprof" + profiles.grafana.com/goroutine.scrape: "true" + profiles.grafana.com/goroutine.port_name: "pprof" + profiles.grafana.com/block.scrape: "true" + profiles.grafana.com/block.port_name: "pprof" + profiles.grafana.com/mutex.scrape: "true" + profiles.grafana.com/mutex.port_name: "pprof" +``` + +| Annotation | Purpose | +|------------|---------| +| `profiles.grafana.com/service_name` | Service name in Pyroscope (`porch-server`, `porch-function-runner`, or `porch-controllers`) | +| `profiles.grafana.com/cpu.scrape` | Scrape CPU profiles (`/debug/pprof/profile`) | +| `profiles.grafana.com/memory.scrape` | Scrape heap profiles (`/debug/pprof/heap`) | +| `profiles.grafana.com/goroutine.scrape` | Scrape goroutine profiles (`/debug/pprof/goroutine`) | +| `profiles.grafana.com/block.scrape` | Scrape block profiles (`/debug/pprof/block`) | +| `profiles.grafana.com/mutex.scrape` | Scrape mutex profiles (`/debug/pprof/mutex`) | +| `profiles.grafana.com/*/port_name` | Named container port to scrape (`pprof`) | + +### Profiles Available from the Pprof Server + +| Endpoint | Profile | Scraped by Alloy | +|----------|---------|------------------| +| `/debug/pprof/profile` | CPU | yes (`cpu`) | +| `/debug/pprof/heap` | Heap memory | yes (`memory`) | +| `/debug/pprof/goroutine` | Goroutine stacks | yes (`goroutine`) | +| `/debug/pprof/block` | Blocking (off-CPU) | yes (`block`) | +| `/debug/pprof/mutex` | Mutex contention | yes (`mutex`) | +| `/debug/pprof/allocs` | Memory allocations | no (available via HTTP) | +| `/debug/pprof/threadcreate` | Thread creation | no (available via HTTP) | +| `/debug/pprof/trace` | Execution trace | no (available via HTTP) | +| `/debug/pprof/cmdline` | Process command line | no (available via HTTP) | +| `/debug/pprof/symbol` | Symbol lookup | no (available via HTTP) | +| `/debug/pprof/` | Index of all profiles | no (available via HTTP) | + +Alloy writes the five scraped profile types into Pyroscope. View flame graphs in the Pyroscope UI (http://localhost:4040 when using the local stack) or the **Pyroscope – Porch profiling** Grafana dashboard. See [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}) for URLs and dashboards. + +### Complete Pprof Example + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: porch-server + namespace: porch-system +spec: + replicas: 1 + selector: + matchLabels: + app: porch-server + template: + metadata: + labels: + app: porch-server + annotations: + prometheus.io/scrape: "true" + prometheus.io/port: "9464" + prometheus.io/path: "/metrics" + profiles.grafana.com/service_name: "porch-server" + profiles.grafana.com/cpu.scrape: "true" + profiles.grafana.com/cpu.port_name: "pprof" + profiles.grafana.com/memory.scrape: "true" + profiles.grafana.com/memory.port_name: "pprof" + profiles.grafana.com/goroutine.scrape: "true" + profiles.grafana.com/goroutine.port_name: "pprof" + profiles.grafana.com/block.scrape: "true" + profiles.grafana.com/block.port_name: "pprof" + profiles.grafana.com/mutex.scrape: "true" + profiles.grafana.com/mutex.port_name: "pprof" + spec: + containers: + - name: porch-server + image: porch-server:latest + env: + - name: OTEL_METRICS_EXPORTER + value: "prometheus" + - name: OTEL_EXPORTER_PROMETHEUS_HOST + value: "0.0.0.0" + - name: OTEL_EXPORTER_PROMETHEUS_PORT + value: "9464" + - name: PORCH_PPROF_PORT + value: "8080" + ports: + - name: metrics + containerPort: 9464 + protocol: TCP + - name: pprof + containerPort: 8080 + protocol: TCP +``` + +Repeat the same `PORCH_PPROF_PORT`, named `pprof` port, and `profiles.grafana.com/*` annotations on `function-runner` (service name `porch-function-runner`) and `porch-controllers` (service name `porch-controllers`). + ## Metrics Configuration ### OTLP Metrics Export @@ -193,7 +343,7 @@ env: ### Prometheus Metrics Export -Porch supports native Prometheus metrics export through an HTTP endpoint. This is the recommended approach for Kubernetes environments with Prometheus-based monitoring. +Porch supports native Prometheus metrics export through an HTTP endpoint. This is the recommended approach for Kubernetes environments with Prometheus-based monitoring. Default Kind manifests already set these variables on porch-server, function-runner, and porch-controllers. #### Basic Prometheus Configuration @@ -225,7 +375,6 @@ All environment variables apply to all Porch components: porch-server, porch-con ## Prometheus Auto-Discovery - ### Pod Annotations (Prometheus Kubernetes SD) For Prometheus using Kubernetes service discovery with pod annotations: @@ -257,9 +406,13 @@ spec: protocol: TCP ``` +The local monitoring stack scrapes Porch Services on port 9464 directly rather than relying on these annotations. See [What Prometheus Scrapes]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment#what-prometheus-scrapes" %}}). + ## Complete Deployment Examples -### Porch Server with OTLP Export (All Signals) +The examples below combine Prometheus metrics, optional OTLP traces, Prometheus scrape annotations, and pprof/Pyroscope annotations. Adjust exporters to match your backend. + +### Porch Server with Prometheus, OTLP traces, and pprof ```yaml apiVersion: apps/v1 @@ -276,24 +429,52 @@ spec: metadata: labels: app: porch-server + annotations: + prometheus.io/scrape: "true" + prometheus.io/port: "9464" + prometheus.io/path: "/metrics" + profiles.grafana.com/service_name: "porch-server" + profiles.grafana.com/cpu.scrape: "true" + profiles.grafana.com/cpu.port_name: "pprof" + profiles.grafana.com/memory.scrape: "true" + profiles.grafana.com/memory.port_name: "pprof" + profiles.grafana.com/goroutine.scrape: "true" + profiles.grafana.com/goroutine.port_name: "pprof" + profiles.grafana.com/block.scrape: "true" + profiles.grafana.com/block.port_name: "pprof" + profiles.grafana.com/mutex.scrape: "true" + profiles.grafana.com/mutex.port_name: "pprof" spec: containers: - name: porch-server image: porch-server:latest env: - - name: OTEL_TRACES_EXPORTER - value: "otlp" + - name: OTEL_SERVICE_NAME + value: "porch-server" - name: OTEL_METRICS_EXPORTER + value: "prometheus" + - name: OTEL_EXPORTER_PROMETHEUS_HOST + value: "0.0.0.0" + - name: OTEL_EXPORTER_PROMETHEUS_PORT + value: "9464" + - name: OTEL_TRACES_EXPORTER value: "otlp" - - name: OTEL_LOGS_EXPORTER - value: "otlp" - - name: OTEL_EXPORTER_OTLP_ENDPOINT + - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT value: "http://otel-collector.observability:4318" - - name: OTEL_EXPORTER_OTLP_PROTOCOL + - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL value: "http/protobuf" + - name: PORCH_PPROF_PORT + value: "8080" + ports: + - name: metrics + containerPort: 9464 + protocol: TCP + - name: pprof + containerPort: 8080 + protocol: TCP ``` -### Porch Controllers with Prometheus Metrics and OTLP Traces +### Porch Controllers with Prometheus Metrics, OTLP Traces, and pprof ```yaml apiVersion: apps/v1 @@ -314,6 +495,17 @@ spec: prometheus.io/scrape: "true" prometheus.io/port: "9464" prometheus.io/path: "/metrics" + profiles.grafana.com/service_name: "porch-controllers" + profiles.grafana.com/cpu.scrape: "true" + profiles.grafana.com/cpu.port_name: "pprof" + profiles.grafana.com/memory.scrape: "true" + profiles.grafana.com/memory.port_name: "pprof" + profiles.grafana.com/goroutine.scrape: "true" + profiles.grafana.com/goroutine.port_name: "pprof" + profiles.grafana.com/block.scrape: "true" + profiles.grafana.com/block.port_name: "pprof" + profiles.grafana.com/mutex.scrape: "true" + profiles.grafana.com/mutex.port_name: "pprof" spec: containers: - name: porch-controllers @@ -333,10 +525,15 @@ spec: value: "http://otel-collector.observability:4318" - name: OTEL_EXPORTER_OTLP_PROTOCOL value: "http/protobuf" + - name: PORCH_PPROF_PORT + value: "8080" ports: - name: metrics containerPort: 9464 protocol: TCP + - name: pprof + containerPort: 8080 + protocol: TCP ``` ### Function Runner with Mixed Configuration @@ -359,6 +556,18 @@ spec: annotations: prometheus.io/scrape: "true" prometheus.io/port: "9464" + prometheus.io/path: "/metrics" + profiles.grafana.com/service_name: "porch-function-runner" + profiles.grafana.com/cpu.scrape: "true" + profiles.grafana.com/cpu.port_name: "pprof" + profiles.grafana.com/memory.scrape: "true" + profiles.grafana.com/memory.port_name: "pprof" + profiles.grafana.com/goroutine.scrape: "true" + profiles.grafana.com/goroutine.port_name: "pprof" + profiles.grafana.com/block.scrape: "true" + profiles.grafana.com/block.port_name: "pprof" + profiles.grafana.com/mutex.scrape: "true" + profiles.grafana.com/mutex.port_name: "pprof" spec: containers: - name: function-runner @@ -378,15 +587,21 @@ spec: value: "http://otel-collector.observability:4318" - name: OTEL_EXPORTER_OTLP_PROTOCOL value: "http/protobuf" + - name: PORCH_PPROF_PORT + value: "8080" ports: - name: metrics containerPort: 9464 + - name: pprof + containerPort: 8080 ``` ### Wrapper Server Configuration via Pod Templating The wrapper-server component can be configured with OpenTelemetry settings through the pod templating mechanism used by the function runner. This is done by creating a ConfigMap with a pod template that includes the necessary environment variables. +wrapper-server does not expose pprof. Configure Prometheus metrics and optional OTLP traces only. + #### ConfigMap Pod Template with OpenTelemetry Configuration ```yaml @@ -504,43 +719,97 @@ env: This allows routing different telemetry signals to specialized backends. - ## Available Metrics -Porch records the following metrics via OpenTelemetry: +Porch records the following metrics via OpenTelemetry. They are emitted by porch-server and porch-controllers (and, where applicable, function-runner). View them on the **Porch API** and **Porch File-system Resources** Grafana dashboards when using the [local monitoring stack]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). + +Load-test process metrics (`porch_perf_*`) are documented separately in [Performance Tests]({{% relref "/docs/12_contributing/code-contribution/performance-tests#performance-test-metrics" %}}). + +### API Call Duration + +| Metric Name | Type | Unit | Description | +|-------------|------|------|-------------| +| `porch_api_call_duration_seconds` | Histogram | seconds | Duration of Porch API calls and related operations | + +| Attribute | Description | +|-----------|-------------| +| `resource` | Resource kind (see table below) | +| `verb` | Operation name (see table below) | +| `api_version` | `v1alpha1` or `v1alpha2`. Omitted for `ExternalRepo` operations, which are shared infrastructure | + +Histogram buckets start at 0.001 seconds and double for 16 boundaries (up to 32.768 seconds). + +### Request Counts + +| Metric Name | Type | Unit | Description | +|-------------|------|------|-------------| +| `porch_api_requests_by_user` | Counter | — | Total number of requests, broken down by resource, operation, and user | + +| Attribute | Description | +|-----------|-------------| +| `resource` | Resource kind (see table below) | +| `op` | Operation name (same values as `verb` on the duration histogram) | +| `user` | Kubernetes user from the request context. v1alpha2 controller operations use `packagerevision-controller` | +| `api_version` | `v1alpha1` or `v1alpha2`. Omitted for `ExternalRepo` operations | + +### Resources and Operations + +| `resource` | Component | `verb` / `op` values | `api_version` | +|------------|-----------|----------------------|---------------| +| `PackageRevision` | porch-server | `LIST`, `GET`, `CREATE`, `UPDATE`, `DELETE` | `v1alpha1` | +| `PackageRevision` | porch-controllers | `CREATE`, `UPDATE`, `DELETE` | `v1alpha2` | +| `PackageRevisionResources` | porch-server | `LIST`, `GET`, `UPDATE` | `v1alpha1` | +| `PackageRevisionResources` | porch-controllers | `UPDATE` | `v1alpha2` | +| `PackageRevisionApproval` | porch-server | `GET`, `UPDATE` | `v1alpha1` | +| `ExternalRepo` | porch-server (git) | `FETCH`, `PUSH` | omitted | ### Package Size Metrics -| Metric Name | Type | Unit | Description | -|----------------------------------|-----------|-------|-------------| -| `porch_package_size_bytes` | Histogram | Bytes | File size of a package's resources expressed as a histogram | -| `porch_package_size_bytes_total` | Gauge | Bytes | Total file size of a package's resources | +| Metric Name | Type | Unit | Description | +|-------------|------|------|-------------| +| `porch_package_size_bytes` | Histogram | Bytes | File size of a package's resources expressed as a histogram | +| `porch_package_size_bytes_total` | Gauge | Bytes | Total file size of a package's resources | Package size metrics are recorded with the following attributes from the relevant package: -| Attribute | Description | -|------------------|-------------| -| `namespace` | Kubernetes namespace of the package revision | -| `repository` | Name of the repository containing the package | -| `package` | Path and name of the package | +| Attribute | Description | +|-----------|-------------| +| `namespace` | Kubernetes namespace of the package revision | +| `repository` | Name of the repository containing the package | +| `package` | Path and name of the package | | `workspace_name` | WorkspaceName of the package revision - short, unique description of the changes | These metrics are recorded as part of every flow that updates package revision resources: + - Create package revision - Delete package revision - Discover/sync package revisions from a registered repository - Delete package revisions on unregistering a repository - Direct update of PackageRevisionResources (e.g. `rpkg push`) -**Prometheus metric names:** +Histogram buckets start at 0, then 1024 bytes, doubling for 21 further boundaries. + +### Prometheus Metric Names When using the Prometheus exporter, these are made available under the metric names: + +**API call duration** + +- `porch_api_call_duration_seconds_bucket` +- `porch_api_call_duration_seconds_count` +- `porch_api_call_duration_seconds_sum` + +**Request counts** + +- `porch_api_requests_by_user_total` + +**Package size** + - `porch_package_size_bytes_bucket` - `porch_package_size_bytes_count` - `porch_package_size_bytes_sum` - `porch_package_size_bytes_total` - ## Troubleshooting ### Verify Metrics Endpoint @@ -552,13 +821,29 @@ kubectl port-forward -n porch-system deployment/porch-server 9464:9464 curl http://localhost:9464/metrics ``` +### Verify Pprof Endpoint + +The pprof server listens only when `PORCH_PPROF_PORT` is set: + +```bash +kubectl port-forward -n porch-system deployment/porch-server 8080:8080 +curl http://localhost:8080/debug/pprof/ +``` + ## Additional Resources - [OpenTelemetry Autoexport Documentation](https://pkg.go.dev/go.opentelemetry.io/contrib/exporters/autoexport) - [OpenTelemetry Environment Variables Specification](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/) +- [Go pprof](https://pkg.go.dev/net/http/pprof) +- [Grafana Alloy profiling annotations](https://grafana.com/docs/alloy/latest/collect/profiles-via-annotation/) + +## See Also + +- [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}) — local Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy +- [Performance Tests]({{% relref "/docs/12_contributing/code-contribution/performance-tests" %}}) — load tests and `porch_perf_*` metrics --- {{% alert title="Note" color="primary" %}} -The autoexport package automatically handles exporter lifecycle for traces and metrics, including graceful shutdown when the application context is cancelled. All environment variables documented here apply to all Porch components: porch-server, porch-controllers, function-runner, and wrapper-server. +The autoexport package automatically handles exporter lifecycle for traces and metrics, including graceful shutdown when the application context is cancelled. All OpenTelemetry environment variables documented here apply to all Porch components: porch-server, porch-controllers, function-runner, and wrapper-server. `PORCH_PPROF_PORT` applies to porch-server, porch-controllers, and function-runner only. {{% /alert %}} diff --git a/docs/content/en/docs/6_configuration_and_deployments/deployments/_index.md b/docs/content/en/docs/6_configuration_and_deployments/deployments/_index.md index 9a5e0bc34..ff8de4306 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/deployments/_index.md +++ b/docs/content/en/docs/6_configuration_and_deployments/deployments/_index.md @@ -16,6 +16,12 @@ Set up Porch locally using Kind for development and testing. This includes: - Choice between CR cache or database cache - Pre-configured development tools +### [Local Performance Monitoring Deployment]({{% relref "local-performance-monitoring-deployment" %}}) +Deploy Prometheus, Grafana, Grafana Alloy, Pyroscope, and Jaeger against a local Kind cluster: +- Base deployment for Porch API, Resource Usage and PostgreSQL metrics visualized in Grafana dashboards +- Optional Jaeger deployment for distributed traces +- Optional Pyroscope and Grafana Alloy deployment for Go CPU, memory, goroutine, block, and mutex profiling + ### [Catalog Deployment]({{% relref "catalog-deployment" %}}) Deploy Porch in production environments using the Nephio catalog: - Production-ready configuration @@ -29,6 +35,11 @@ Deploy Porch in production environments using the Nephio catalog: - Learning Porch concepts - Contributing to Porch development +- **Use Local Performance Monitoring** when: + - Measuring API latency, resource usage, or package sizes + - Running [performance tests]({{% relref "/docs/12_contributing/code-contribution/performance-tests" %}}) + - Inspecting traces or Go profiles locally + - **Use Catalog Deployment** when: - Deploying to production environments - Integrating with existing infrastructure diff --git a/docs/content/en/docs/6_configuration_and_deployments/deployments/catalog-deployment.md b/docs/content/en/docs/6_configuration_and_deployments/deployments/catalog-deployment.md index 26f0d9fbb..52ca08a48 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/deployments/catalog-deployment.md +++ b/docs/content/en/docs/6_configuration_and_deployments/deployments/catalog-deployment.md @@ -27,7 +27,7 @@ If using **DB Cache**, you must configure database settings for **both** Porch S These **optional** features must be configured **before** deployment if you need them: #### Porch Server -- [OpenTelemetry]({{% relref "../configurations/opentelemetry" %}}) - Enable distributed tracing and metrics (requires deployment env vars) +- [OpenTelemetry]({{% relref "../configurations/opentelemetry" %}}) - Enable distributed tracing, metrics, and pprof (requires deployment env vars and annotations) - [Git Custom TLS]({{% relref "../configurations/components/porch-server-config/git-authentication#3-httpstls-configuration" %}}) - Enable custom TLS certificates for Git repositories (requires `--use-git-cabundle=true` arg) #### Porch Controllers diff --git a/docs/content/en/docs/6_configuration_and_deployments/deployments/local-dev-env-deployment.md b/docs/content/en/docs/6_configuration_and_deployments/deployments/local-dev-env-deployment.md index 24d5a1a03..288775818 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/deployments/local-dev-env-deployment.md +++ b/docs/content/en/docs/6_configuration_and_deployments/deployments/local-dev-env-deployment.md @@ -103,4 +103,6 @@ kind delete cluster --name porch-test ## Next Steps - Follow the [Getting Started tutorial]({{% relref "/docs/3_getting_started" %}}) to create your first packages +- Deploy [local performance monitoring]({{% relref "local-performance-monitoring-deployment" %}}) for Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy +- Run [performance tests]({{% relref "/docs/12_contributing/code-contribution/performance-tests" %}}) against this Kind cluster - See [Development Process]({{% relref "/docs/12_contributing" %}}) for contributing guidelines diff --git a/docs/content/en/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment.md b/docs/content/en/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment.md new file mode 100644 index 000000000..8e3bc823f --- /dev/null +++ b/docs/content/en/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment.md @@ -0,0 +1,125 @@ +--- +title: "Local Performance Monitoring Deployment" +type: docs +weight: 4 +description: Deploy Prometheus, Grafana, Grafana Alloy, Pyroscope and Jaeger for local Porch performance monitoring +--- + +This guide deploys a local observability stack against a Kind based Porch cluster. Use it to inspect API metrics, resource usage, traces, profiles, and [performance test]({{% relref "/docs/12_contributing/code-contribution/performance-tests" %}}) results. + +{{% alert title="Note" color="primary" %}} +This stack is intended for local development and performance testing. For production telemetry, configure exporters directly on Porch components as described in [OpenTelemetry Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}). +{{% /alert %}} + +## Prerequisites + +- A running Kind cluster with Porch deployed. See [Local Development Environment]({{% relref "local-dev-env-deployment" %}}). +- [kubectl](https://kubernetes.io/docs/tasks/tools/) +- [kpt CLI](https://kpt.dev/installation/) +- [make](https://www.gnu.org/software/make/) + +Verify Porch is running before deploying monitoring: + +```bash +kubectl get pods -n porch-system +``` + +## Deploy the Monitoring Stack + +All make targets run from the repository root. Deployments are created in the `porch-monitoring` namespace. Port-forwarding starts automatically after a successful deploy. + +### Base Stack + +```bash +make deploy-monitoring +``` + +Creates `porch-monitoring` and deploys: + +| Component | Purpose | +|-----------|---------| +| **Prometheus** | Scrapes Porch component metrics on port 9464, PostgreSQL metrics from postgres-exporter, cAdvisor container metrics, and performance-test metrics on host `172.17.0.1:9095` | +| **Grafana** | Pre-loaded with Porch dashboards (see [Grafana Dashboards](#grafana-dashboards)) | +| **Postgres Exporter** | Exports PostgreSQL metrics from `porch-postgresql` (used when Porch runs with [DB cache]({{% relref "/docs/6_configuration_and_deployments/configurations/cache" %}})) | + +### Optional: Jaeger + +```bash +make deploy-monitoring-jaeger +``` + +Deploys Jaeger and enables OTLP trace export on `porch-server`, `function-runner`, and `porch-controllers`. Trace environment variables are documented in [OpenTelemetry Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}). + +### Optional: Pyroscope and Alloy + +```bash +make deploy-monitoring-pyroscope +``` + +Deploys the continuous-profiling stack: + +| Component | Purpose | +|-----------|---------| +| **Pyroscope** | Stores profiles and serves the UI at http://localhost:4040 | +| **Grafana Alloy** | Discovers Porch pods, scrapes their pprof endpoints, and writes profiles into Pyroscope | + +Alloy has no UI of its own. It watches pods in the cluster, keeps those that have `profiles.grafana.com/*` annotations, scrapes the named `pprof` container port, and forwards CPU, memory (heap), goroutine, block, and mutex profiles to Pyroscope at `http://pyroscope.porch-monitoring.svc.cluster.local:4040`. + +`make deploy-monitoring-pyroscope` also enables pprof on `porch-server`, `function-runner`, and `porch-controllers` by setting `PORCH_PPROF_PORT` and adding the Alloy discovery annotations. The environment variable, annotations, and available profiles are documented in [Pprof Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry#pprof-configuration" %}}). + +### Cleanup and Restart + +```bash +make cleanup-monitoring # remove monitoring resources and disable trace/pprof export +make restart-monitoring # cleanup, then redeploy the base stack +``` + +## Access the UIs + +| Service | URL | Notes | +|---------|-----|-------| +| Prometheus | http://localhost:9092 | Query raw metrics | +| Grafana | http://localhost:3001 | Dashboards listed below | +| Jaeger | http://localhost:16686 | After `make deploy-monitoring-jaeger` | +| Pyroscope | http://localhost:4040 | After `make deploy-monitoring-pyroscope`. Alloy scrapes Porch pods and writes profiles here; Alloy itself has no UI | + +Grafana credentials are printed when the stack is deployed. The default username is `porch`. The password is stored in the `grafana-admin-creds` secret: + +```bash +kubectl -n porch-monitoring get secret grafana-admin-creds \ + -o jsonpath='{.data.GF_SECURITY_ADMIN_PASSWORD}' | base64 -d +echo +``` + +## Grafana Dashboards + +Open Grafana at http://localhost:3001 and select a dashboard. The following dashboards are loaded automatically: + +| Dashboard | What it shows | +|-----------|----------------| +| **Porch API** | API call latency (p95, p99, average) for PackageRevision, PackageRevisionResources, and Approval operations; ExternalRepo FETCH/PUSH latency; request rates by user | +| **Porch Resource Usage** | CPU and memory for porch-server, function-runner, porch-controllers, PostgreSQL, and function pods; controller workqueue depth, reconcile rate, and p99 reconcile duration | +| **Porch PostgreSQL** | Query activity, row operations, QPS, cache hit ratio, connections, buffers, and conflicts/deadlocks (DB cache deployments) | +| **Porch File-system Resources** | Package revision resource sizes (`porch_package_size_bytes_total`) | +| **Porch Performance Test Dashboard** | Per-operation duration, count, and success rate from performance tests (`porch_perf_*` metrics) | +| **Pyroscope – Porch profiling** | Flame graphs for CPU, memory, goroutine, block, and mutex profiles that Alloy scraped from Porch pods | + +Server-side metric names and attributes are listed in [Available Metrics]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry#available-metrics" %}}). Performance-test metric names are listed in [Performance Tests]({{% relref "/docs/12_contributing/code-contribution/performance-tests#performance-test-metrics" %}}). + +## What Prometheus Scrapes + +| Target | Address | Metrics | +|--------|---------|---------| +| porch-server | `api.porch-system.svc.cluster.local:9464` | Porch API and package-size metrics | +| porch-controllers | `porch-controllers.porch-system.svc.cluster.local:9464` | Controller API metrics and controller-runtime metrics | +| function-runner | `function-runner.porch-system.svc.cluster.local:9464` | Function-runner metrics | +| postgres-exporter | `postgres-exporter.porch-monitoring.svc.cluster.local:9187` | PostgreSQL metrics | +| kubelet cAdvisor | Kubernetes node proxy | Container CPU and memory | +| Performance tests | `172.17.0.1:9095` | `porch_perf_*` metrics when tests run with `-enable-prometheus=true` | + +The performance-test process listens on **port 9095** on the host. Prometheus inside Kind reaches it through the Docker bridge gateway (`172.17.0.1`). Only set `-enable-prometheus=true` when this stack is deployed; the test waits 15 seconds before shutting down its metrics server so Prometheus can scrape final values. + +## Next Steps + +- Configure exporters, pprof, and metric names: [OpenTelemetry Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}) +- Run load and soak tests: [Performance Tests]({{% relref "/docs/12_contributing/code-contribution/performance-tests" %}}) From de9e0444e4a131fbfc074667842caf0ddd3a0080 Mon Sep 17 00:00:00 2001 From: Rendre Greyling Date: Wed, 9 Sep 2026 14:05:55 +0200 Subject: [PATCH 2/5] First cleanup Signed-off-by: Rendre Greyling --- .../configurations/_index.md | 4 ++-- .../configurations/components/_index.md | 4 ---- .../function-runner-config/_index.md | 6 +----- .../components/porch-server-config/_index.md | 2 +- .../configurations/opentelemetry.md | 19 +++++++++---------- 5 files changed, 13 insertions(+), 22 deletions(-) diff --git a/docs/content/en/docs/6_configuration_and_deployments/configurations/_index.md b/docs/content/en/docs/6_configuration_and_deployments/configurations/_index.md index 65d666f67..b74969036 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/configurations/_index.md +++ b/docs/content/en/docs/6_configuration_and_deployments/configurations/_index.md @@ -20,9 +20,9 @@ Configure individual Porch components: ### OTEL Metrics, Tracing, and Profiling -[OpenTelemetry]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}) - Tracing, metrics, and pprof configuration +[OpenTelemetry]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}) - Tracing, metrics, and profiling configuration -For a local Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy stack, see [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). +[Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}) - Local Prometheus, Grafana, Grafana Alloy, Jaeger and Pyroscope observability stack. ## Core Configuration Options diff --git a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/_index.md b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/_index.md index 5d0e3f69e..00bd85412 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/_index.md +++ b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/_index.md @@ -22,7 +22,3 @@ Manage the lifecycle of Repositories, PackageRevisions, PackageVariants, and Pac ### [Function Runner]({{% relref "function-runner-config" %}}) Executes KRM functions in isolated containers: - [Private Registry Access]({{% relref "function-runner-config/private-registries-config" %}}) - Container registry authentication - -## Observability - -[OpenTelemetry]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}) covers metrics, traces, and pprof for every component. Deploy the local stack (including Pyroscope and Grafana Alloy) with [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). \ No newline at end of file diff --git a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/function-runner-config/_index.md b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/function-runner-config/_index.md index e09f03890..7ec84a91c 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/function-runner-config/_index.md +++ b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/function-runner-config/_index.md @@ -194,12 +194,8 @@ spec: periodSeconds: 5 ``` -## Observability - -For tracing, metrics, and pprof configuration, see [OpenTelemetry Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}). For a local Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy stack, see [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). - {{% alert title="Note" color="primary" %}} For advanced configuration options: - [Pod Templates]({{% relref "pod-templates" %}}) - Customize function pod specifications - [Private Registries]({{% relref "private-registries-config" %}}) - Configure private registry access -{{% /alert %}} \ No newline at end of file +{{% /alert %}} diff --git a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-server-config/_index.md b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-server-config/_index.md index 887b12fc2..4dcb0df75 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-server-config/_index.md +++ b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-server-config/_index.md @@ -102,4 +102,4 @@ readinessProbe: port: 8080 initialDelaySeconds: 5 periodSeconds: 5 -``` \ No newline at end of file +``` diff --git a/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md b/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md index 1000c8ec0..a14b6b3ad 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md +++ b/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md @@ -2,7 +2,7 @@ title: "OpenTelemetry Configuration" type: docs weight: 4 -description: Configure OpenTelemetry metrics, traces, and pprof profiling for Porch components +description: Configure OpenTelemetry metrics, traces export, and profiling for Porch components --- ## Overview @@ -11,15 +11,16 @@ Porch supports OpenTelemetry observability through the [autoexport package](http All Porch components (porch-server, porch-controllers, function-runner, and wrapper-server) support OpenTelemetry configuration through standardized environment variables as defined by the [OpenTelemetry specification](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/). -Porch also exposes Go [pprof](https://pkg.go.dev/net/http/pprof) endpoints for continuous profiling. Grafana Alloy can scrape those endpoints into Pyroscope when the pod annotations and `PORCH_PPROF_PORT` environment variable described in [Pprof Configuration](#pprof-configuration) are set. +Porch also exposes Go [pprof](https://pkg.go.dev/net/http/pprof) endpoints for continuous profiling. Grafana Alloy can scrape those endpoints into Pyroscope when the pod annotations and `PORCH_PPROF_PORT` environment variable described in [Profiling Configuration](#profiling-configuration) are set. Default Kind deployments already export Prometheus metrics on port 9464 (`OTEL_METRICS_EXPORTER=prometheus`) and declare a container port named `pprof`. Trace export and the pprof HTTP server stay disabled until you set the corresponding environment variables. +For a local monitoring stack, see [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). For load-test metrics emitted by the performance test process, see [Performance Tests]({{% relref "/docs/12_contributing/code-contribution/performance-tests" %}}). + {{% alert title="Note" color="primary" %}} **Current Implementation Status**: Porch currently implements metrics and traces export. Logs export is not supported. {{% /alert %}} -For a local Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy stack, see [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). For load-test metrics emitted by the test process, see [Performance Tests]({{% relref "/docs/12_contributing/code-contribution/performance-tests" %}}). ## Traces Configuration @@ -117,8 +118,6 @@ kubectl port-forward -n porch-system service/jaeger-http 16686 Open http://localhost:16686 and you should see `porch-server`, `porch-function-runner`, and `porch-controllers` in the service dropdown. -To deploy Jaeger as part of the local monitoring stack (namespace `porch-monitoring`) instead of applying the manifest above, see [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). - ### OTLP Trace Export Export traces to an OpenTelemetry Protocol (OTLP) collector using either HTTP or gRPC protocols. @@ -169,15 +168,15 @@ All environment variables apply to all Porch components: porch-server, porch-con | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | Traces-specific endpoint (overrides general endpoint) | - | `http://localhost:4318/v1/traces` | | `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` | Traces-specific protocol (overrides general protocol) | - | `http/protobuf`, `grpc` | -## Pprof Configuration +## Profiling Configuration -Porch components start a pprof HTTP server when `PORCH_PPROF_PORT` is set. Grafana Alloy uses `profiles.grafana.com/*` pod annotations to discover those endpoints and ship profiles to Pyroscope. +Porch components start a Go pprof HTTP server when `PORCH_PPROF_PORT` is set. Grafana Alloy uses `profiles.grafana.com/*` pod annotations to discover those endpoints and ship profiles to Pyroscope. {{% alert title="Warning" color="warning" %}} Enabling pprof turns on mutex and block profiling (`runtime.SetMutexProfileFraction(1)` and `runtime.SetBlockProfileRate(1)`). That adds overhead. Use it in development and performance-test clusters, not as a default in production, unless you have measured the cost. {{% /alert %}} -Pprof applies to porch-server, function-runner, and porch-controllers. wrapper-server does not expose a pprof server. +Profiling applies to porch-server, function-runner, and porch-controllers. ### Environment Variable @@ -253,7 +252,7 @@ metadata: | `/debug/pprof/symbol` | Symbol lookup | no (available via HTTP) | | `/debug/pprof/` | Index of all profiles | no (available via HTTP) | -Alloy writes the five scraped profile types into Pyroscope. View flame graphs in the Pyroscope UI (http://localhost:4040 when using the local stack) or the **Pyroscope – Porch profiling** Grafana dashboard. See [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}) for URLs and dashboards. +Alloy writes the five scraped profile types into Pyroscope where it can be viewed in the Pyroscope or Grafana UI. ### Complete Pprof Example @@ -343,7 +342,7 @@ env: ### Prometheus Metrics Export -Porch supports native Prometheus metrics export through an HTTP endpoint. This is the recommended approach for Kubernetes environments with Prometheus-based monitoring. Default Kind manifests already set these variables on porch-server, function-runner, and porch-controllers. +Porch supports native Prometheus metrics export through an HTTP endpoint. This is the recommended approach for Kubernetes environments with Prometheus-based monitoring. #### Basic Prometheus Configuration From 793ec50dbc7d9d619d938740fa730efb49ad07ab Mon Sep 17 00:00:00 2001 From: Rendre Greyling Date: Wed, 9 Sep 2026 14:28:16 +0200 Subject: [PATCH 3/5] Further cleanup Signed-off-by: Rendre Greyling --- .../12_contributing/code-contribution/performance-tests.md | 6 +++--- .../components/porch-controllers-config/_index.md | 7 +------ .../configurations/opentelemetry.md | 5 ++--- .../deployments/catalog-deployment.md | 2 +- .../deployments/local-dev-env-deployment.md | 2 +- 5 files changed, 8 insertions(+), 14 deletions(-) diff --git a/docs/content/en/docs/12_contributing/code-contribution/performance-tests.md b/docs/content/en/docs/12_contributing/code-contribution/performance-tests.md index a04104f72..14739242c 100644 --- a/docs/content/en/docs/12_contributing/code-contribution/performance-tests.md +++ b/docs/content/en/docs/12_contributing/code-contribution/performance-tests.md @@ -5,7 +5,7 @@ weight: 2 description: Run Porch package revision lifecycle performance and load tests --- -Performance tests measure Porch package revision lifecycle latency and throughput under configurable load. They create Gitea and Porch repositories, run package revisions through the full lifecycle (create → update resources → propose → approve/publish), optionally delete them, and record per-operation timings plus optional Prometheus metrics. +Performance tests measure Porch package revision lifecycle latency and throughput under configurable load. They create Gitea and Porch repositories in a local kind cluster. Then run package revisions through the full lifecycle (create → update resources → propose → approve/publish), optionally delete them, and record per-operation timings plus optional Prometheus metrics. For the local Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy stack used with these tests, see [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). Server-side metrics and pprof configuration are documented in [OpenTelemetry Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}). @@ -99,7 +99,7 @@ Tests are gated by environment variables and skipped otherwise. Run from the rep cd test/performance/ ``` -### Scale / Load Test (`TestPorchScalePerformance`) +### Load Test Creates a configured number of repositories, packages, and revisions to simulate load. Runs repositories and packages in parallel up to the configured parallelism limits. @@ -124,7 +124,7 @@ LOAD_TEST=1 go test -v ./... \ -timeout 2h ``` -### Maximum Package Revisions Test (`TestIncreasePRsPerformance`) +### Maximum Package Revisions Test Creates package revisions sequentially in a single repository until the error-rate threshold is exceeded. Designed for long-running soak tests. diff --git a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-controllers-config/_index.md b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-controllers-config/_index.md index 1bd19a549..12f25a48e 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-controllers-config/_index.md +++ b/docs/content/en/docs/6_configuration_and_deployments/configurations/components/porch-controllers-config/_index.md @@ -125,9 +125,4 @@ Both the PackageRevision and Repository webhooks run in the porch-controllers po For webhook TLS certificate setup and management, see [Webhook Certificate Management](../porch-webhooks/cert-manager-webhooks.md). -For details on webhook validation rules, see [Webhook Validation Rules](../porch-webhooks/validation-rules.md). - - -## Observability - -For tracing, metrics, and pprof configuration, see [OpenTelemetry Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry" %}}). For a local Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy stack, see [Local Performance Monitoring Deployment]({{% relref "/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment" %}}). \ No newline at end of file +For details on webhook validation rules, see [Webhook Validation Rules](../porch-webhooks/validation-rules.md). \ No newline at end of file diff --git a/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md b/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md index a14b6b3ad..507ee423e 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md +++ b/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md @@ -21,7 +21,6 @@ For a local monitoring stack, see [Local Performance Monitoring Deployment]({{% **Current Implementation Status**: Porch currently implements metrics and traces export. Logs export is not supported. {{% /alert %}} - ## Traces Configuration ### Quick Start with Jaeger @@ -208,7 +207,7 @@ ports: ### Pod Annotations for Pyroscope -Add these annotations on the pod template so Grafana Alloy scrapes the pprof port: +Add these annotations on the pod template so that Grafana Alloy can scrape the pprof port: ```yaml metadata: @@ -599,7 +598,7 @@ spec: The wrapper-server component can be configured with OpenTelemetry settings through the pod templating mechanism used by the function runner. This is done by creating a ConfigMap with a pod template that includes the necessary environment variables. -wrapper-server does not expose pprof. Configure Prometheus metrics and optional OTLP traces only. +The wrapper-server does not expose pprof. Configure Prometheus metrics and optional OTLP traces only. #### ConfigMap Pod Template with OpenTelemetry Configuration diff --git a/docs/content/en/docs/6_configuration_and_deployments/deployments/catalog-deployment.md b/docs/content/en/docs/6_configuration_and_deployments/deployments/catalog-deployment.md index 52ca08a48..757b312f7 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/deployments/catalog-deployment.md +++ b/docs/content/en/docs/6_configuration_and_deployments/deployments/catalog-deployment.md @@ -27,7 +27,7 @@ If using **DB Cache**, you must configure database settings for **both** Porch S These **optional** features must be configured **before** deployment if you need them: #### Porch Server -- [OpenTelemetry]({{% relref "../configurations/opentelemetry" %}}) - Enable distributed tracing, metrics, and pprof (requires deployment env vars and annotations) +- [OpenTelemetry]({{% relref "../configurations/opentelemetry" %}}) - Enable distributed tracing, metrics, and profiling (requires deployment env vars and annotations) - [Git Custom TLS]({{% relref "../configurations/components/porch-server-config/git-authentication#3-httpstls-configuration" %}}) - Enable custom TLS certificates for Git repositories (requires `--use-git-cabundle=true` arg) #### Porch Controllers diff --git a/docs/content/en/docs/6_configuration_and_deployments/deployments/local-dev-env-deployment.md b/docs/content/en/docs/6_configuration_and_deployments/deployments/local-dev-env-deployment.md index 288775818..99b5f4d5f 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/deployments/local-dev-env-deployment.md +++ b/docs/content/en/docs/6_configuration_and_deployments/deployments/local-dev-env-deployment.md @@ -103,6 +103,6 @@ kind delete cluster --name porch-test ## Next Steps - Follow the [Getting Started tutorial]({{% relref "/docs/3_getting_started" %}}) to create your first packages -- Deploy [local performance monitoring]({{% relref "local-performance-monitoring-deployment" %}}) for Prometheus, Grafana, Jaeger, Pyroscope, and Grafana Alloy +- Deploy [local performance monitoring]({{% relref "local-performance-monitoring-deployment" %}}) for Prometheus, Grafana, Grafana Alloy, Jaeger and Pyroscope - Run [performance tests]({{% relref "/docs/12_contributing/code-contribution/performance-tests" %}}) against this Kind cluster - See [Development Process]({{% relref "/docs/12_contributing" %}}) for contributing guidelines From 2bace6a3e5c3434662c2e090cb1b882b4b6ede66 Mon Sep 17 00:00:00 2001 From: Rendre Greyling Date: Wed, 9 Sep 2026 14:47:32 +0200 Subject: [PATCH 4/5] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Rendre Greyling --- .../configurations/opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md b/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md index 507ee423e..cb15f1aa6 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md +++ b/docs/content/en/docs/6_configuration_and_deployments/configurations/opentelemetry.md @@ -785,7 +785,7 @@ These metrics are recorded as part of every flow that updates package revision r - Delete package revisions on unregistering a repository - Direct update of PackageRevisionResources (e.g. `rpkg push`) -Histogram buckets start at 0, then 1024 bytes, doubling for 21 further boundaries. +Histogram buckets start at 0, then 1024 bytes, doubling for 20 further boundaries. ### Prometheus Metric Names From 6d3f387c40adc884cdbcb1112e48bfb34ffafc50 Mon Sep 17 00:00:00 2001 From: Rendre Greyling Date: Wed, 9 Sep 2026 14:50:47 +0200 Subject: [PATCH 5/5] Address copilot reviews Signed-off-by: Rendre Greyling --- .../deployments/local-dev-env-deployment.md | 2 +- .../deployments/local-performance-monitoring-deployment.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/content/en/docs/6_configuration_and_deployments/deployments/local-dev-env-deployment.md b/docs/content/en/docs/6_configuration_and_deployments/deployments/local-dev-env-deployment.md index 99b5f4d5f..6f70f334b 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/deployments/local-dev-env-deployment.md +++ b/docs/content/en/docs/6_configuration_and_deployments/deployments/local-dev-env-deployment.md @@ -103,6 +103,6 @@ kind delete cluster --name porch-test ## Next Steps - Follow the [Getting Started tutorial]({{% relref "/docs/3_getting_started" %}}) to create your first packages -- Deploy [local performance monitoring]({{% relref "local-performance-monitoring-deployment" %}}) for Prometheus, Grafana, Grafana Alloy, Jaeger and Pyroscope +- Deploy [local performance monitoring]({{% relref "local-performance-monitoring-deployment" %}}) for Prometheus, Grafana, Grafana Alloy, Jaeger and Pyroscope - Run [performance tests]({{% relref "/docs/12_contributing/code-contribution/performance-tests" %}}) against this Kind cluster - See [Development Process]({{% relref "/docs/12_contributing" %}}) for contributing guidelines diff --git a/docs/content/en/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment.md b/docs/content/en/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment.md index 8e3bc823f..888643d59 100644 --- a/docs/content/en/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment.md +++ b/docs/content/en/docs/6_configuration_and_deployments/deployments/local-performance-monitoring-deployment.md @@ -65,7 +65,7 @@ Deploys the continuous-profiling stack: Alloy has no UI of its own. It watches pods in the cluster, keeps those that have `profiles.grafana.com/*` annotations, scrapes the named `pprof` container port, and forwards CPU, memory (heap), goroutine, block, and mutex profiles to Pyroscope at `http://pyroscope.porch-monitoring.svc.cluster.local:4040`. -`make deploy-monitoring-pyroscope` also enables pprof on `porch-server`, `function-runner`, and `porch-controllers` by setting `PORCH_PPROF_PORT` and adding the Alloy discovery annotations. The environment variable, annotations, and available profiles are documented in [Pprof Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry#pprof-configuration" %}}). +`make deploy-monitoring-pyroscope` also enables pprof on `porch-server`, `function-runner`, and `porch-controllers` by setting `PORCH_PPROF_PORT` and adding the Alloy discovery annotations. The environment variable, annotations, and available profiles are documented in [Profiling Configuration]({{% relref "/docs/6_configuration_and_deployments/configurations/opentelemetry#profiling-configuration" %}}). ### Cleanup and Restart