Add opt-in staged startup to flatten the Agent startup memory bump - #53334
Draft
GeorgeHahn wants to merge 4 commits into
Draft
Add opt-in staged startup to flatten the Agent startup memory bump#53334GeorgeHahn wants to merge 4 commits into
GeorgeHahn wants to merge 4 commits into
Conversation
Contributor
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b775e8a6d5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Comment on lines
+168
to
+170
| return deps.StartupSequencer.Defer(startupsequencer.StageIngest, "logs-agent", logsAgent.start) | ||
| }, | ||
| OnStop: logsAgent.stop, |
There was a problem hiding this comment.
When staged_start.enabled is true and logs are enabled, this hook only queues logsAgent.start, so Fx considers the hook started and will still call logsAgent.stop if the Agent is stopped before StageIngest executes. In that window a.schedulers, a.launchers, a.destinationsCtx, and the other fields initialized by start are still nil, but stop unconditionally passes them to SerialStopper and calls a.destinationsCtx.Stop(), which can panic during early shutdown. Gate OnStop on a successful deferred start or make stop tolerate the not-yet-started state.
Useful? React with 👍 / 👎.
Comment on lines
+105
to
+106
| if err := d.fn(ctx); err != nil { | ||
| s.log.Errorf("staged startup: %q (stage %d) failed: %v", d.name, stage, err) |
There was a problem hiding this comment.
With staged startup enabled, deferred OnStart work now runs after Fx startup from this goroutine, but an error is only logged here. For example, logsAgent.start still returns errors for invalid endpoints, and with staging disabled that error aborts startup; with staging enabled the Agent continues and can report as running while the configured subsystem never started. Propagate these deferred startup failures into shutdown/fail-start behavior instead of swallowing them.
Useful? React with 👍 / 👎.
Contributor
Go Package Import DifferencesBaseline: 8d8cc34
|
This comment has been minimized.
This comment has been minimized.
Contributor
Files inventory check summaryFile checks results against ancestor 8d8cc340: Results for datadog-agent_7.83.0~devel.git.357.fe7320b.pipeline.126644327-1_amd64.deb:No change detected |
Contributor
Static quality checks✅ Please find below the results from static quality gates Successful checksInfo
4 successful checks with minimal change (< 2 KiB)
|
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: 8d8cc34 Optimization Goals: ✅ Improvement(s) detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | quality_gate_private_action_runner | memory utilization | -0.01 | [-0.13, +0.11] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_mean_fs_load | memory utilization | -0.86 | [-0.92, -0.80] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle_all_features | memory utilization | -1.03 | [-1.11, -0.95] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle | memory utilization | -1.17 | [-1.25, -1.09] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_metrics_logs | memory utilization | -1.31 | [-1.55, -1.06] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_no_fs_load | memory utilization | -2.06 | [-2.14, -1.98] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_idle | memory utilization | -2.31 | [-2.37, -2.25] | 1 | Logs bounds checks dashboard |
| ✅ | quality_gate_logs | % cpu utilization | -6.78 | [-7.85, -5.71] | 1 | Logs bounds checks dashboard |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | quality_gate_idle | intake_connections | 10/10 | 3 ≤ 4 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 145.66MiB ≤ 154MiB | bounds checks dashboard |
| ✅ | quality_gate_idle | total_bytes_received | 10/10 | 725.43KiB ≤ 819.20KiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 2 ≤ 4 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 495.05MiB ≤ 512MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | total_bytes_received | 10/10 | 1.09MiB ≤ 1.25MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 5 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 185.08MiB ≤ 195MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_logs | total_bytes_received | 10/10 | 239.74MiB ≤ 292MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 374.98 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 5 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 417.07MiB ≤ 430MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | total_bytes_received | 10/10 | 0.88GiB ≤ 1.04GiB | bounds checks dashboard |
| ✅ | quality_gate_private_action_runner | memory_usage | 10/10 | 70.85MiB ≤ 75MiB | bounds checks dashboard |
| ✅ | quality_gate_security_idle | cpu_usage | 10/10 | 30.28 ≤ 100 | bounds checks dashboard |
| ✅ | quality_gate_security_idle | memory_usage | 10/10 | 279.64MiB ≤ 330MiB | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | cpu_usage | 10/10 | 74.77 ≤ 200 | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | memory_usage | 10/10 | 270.94MiB ≤ 310MiB | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | cpu_usage | 10/10 | 24.00 ≤ 100 | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | memory_usage | 10/10 | 261.26MiB ≤ 320MiB | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
Replicate Execution Details
We run multiple replicates for each experiment/variant. However, we allow replicates to be automatically retried if there are any failures, up to 8 times, at which point the replicate is marked dead and we are unable to run analysis for the entire experiment. We call each of these attempts at running replicates a replicate execution. This section lists all replicate executions that failed due to the target crashing or being oom killed.
Note: In the below tables we bucket failures by experiment, variant, and failure type. For each of these buckets we list out the replicate indexes that failed with an annotation signifying how many times said replicate failed with the given failure mode. In the below example the baseline variant of the experiment named experiment_with_failures had two replicates that failed by oom kills. Replicate 0, which failed 8 executions, and replicate 1 which failed 6 executions, all with the same failure mode.
| Experiment | Variant | Replicates | Failure | Logs | Debug Dashboard |
|---|---|---|---|---|---|
| experiment_with_failures | baseline | 0 (x8) 1 (x6) | Oom killed | Debug Dashboard |
The debug dashboard links will take you to a debugging dashboard specifically designed to investigate replicate execution failures.
❌ Retried Profiling Replicate Execution Failures (ddprof)
Note: Profiling replicas may still be executing. See the debug dashboard for up to date status.
| Experiment | Variant | Replicates | Failure | Debug Dashboard |
|---|---|---|---|---|
| quality_gate_idle | baseline | 10 | Oom killed | Debug Dashboard |
| quality_gate_idle_all_features | baseline | 10 | Oom killed | Debug Dashboard |
| quality_gate_idle_all_features | comparison | 10 | Oom killed | Debug Dashboard |
| quality_gate_logs | baseline | 10 | Oom killed | Debug Dashboard |
| quality_gate_logs | comparison | 10 | Oom killed | Debug Dashboard |
| quality_gate_metrics_logs | baseline | 10 | Oom killed | Debug Dashboard |
| quality_gate_metrics_logs | comparison | 10 | Oom killed | Debug Dashboard |
| quality_gate_security_idle | baseline | 10 | Oom killed | Debug Dashboard |
| quality_gate_security_idle | comparison | 10 | Crashed (exit code: 134) | Debug Dashboard |
| quality_gate_security_no_fs_load | baseline | 10 | Crashed (exit code: 134) | Debug Dashboard |
| quality_gate_security_no_fs_load | comparison | 10 | Crashed (exit code: 134) | Debug Dashboard |
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_private_action_runner, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
During boot the core Agent instantiates and starts every subsystem at once: fx runs all OnStart hooks back-to-back, then startAgent loads and schedules all checks. The resulting burst of concurrent allocation makes the Go heap overshoot well past steady state before GC catches up, and that peak becomes the resident high-water-mark that memory regression gates measure. Add a startupsequencer component that lets subsystems register their start work with Defer(stage, ...) instead of running it inline in their OnStart hook. When staged startup is enabled, the sequencer runs the deferred work in ordered stages from a background goroutine, reclaiming transient memory between stages so the peak RSS stays close to steady state. Because fx runs OnStart hooks sequentially, the work must be registered-and-returned rather than blocked on, otherwise later hooks would never run. The feature is opt-in via staged_start.enabled (default false). When disabled, Defer runs the work inline, so behavior is identical to today. The core bundle defaults to a no-op sequencer; core.WithStagedStartup() swaps in the real one for `agent run`. dogstatsd server, the logs agent, and autodiscovery LoadAndRun are migrated as the initial set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Flip staged_start.enabled to true so the staged startup path is exercised by default (and measurable in SMP). It can still be turned off by setting staged_start.enabled to false. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GeorgeHahn
force-pushed
the
hahn/staged-startup
branch
from
July 23, 2026 17:31
b775e8a to
27b3119
Compare
The SMP quality gates warm up for ~45s and measure a steady-state window afterward, with peak memory landing just after warmup. A 5s stage interval finished the staged ramp entirely within the warmup, so every subsystem was already fully started (and had allocated) before measurement began, leaving the memory peak unchanged. Widen the interval to 30s so the ramp extends past the warmup and across the post-warmup peak. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The startup memory peak measured by the SMP quality gates is dominated by the ancillary binaries, not the core Agent: system-probe's eBPF module loading (~19 MiB transient from BTF/bytecode scratch) and the security Agent's runtime-security/compliance init (~16 MiB), whereas the core Agent's own in-window transient is only a few MiB. Apply the same pacing there, gated by the existing staged_start config: - system-probe: load enabled eBPF modules one at a time, reclaiming each module's transient loader scratch and pausing briefly between them (total delay bounded by one stage interval). This runs in the post-app.Start main path, so the pauses don't affect fx's start timeout. - security-agent: reclaim + pause before starting runtime security and again before compliance, so their peaks don't stack (and land later, away from system-probe's early eBPF-load peak). The fx start timeout is 5 minutes via TemporaryAppTimeouts, so the in-Provide pauses are safe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
This pull request has been automatically marked as stale because it has not had activity in the past 15 days. It will be closed in 30 days if no further activity occurs. If this pull request is still relevant, adding a comment or pushing new commits will keep it open. Also, you can always reopen the pull request if you missed the window. Thank you for your contributions! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment with staged startup to reduce the memory usage peak that happens in the first minutes of starting the agent.
(AI below)
What problem does this solve?
At boot the core Agent brings every subsystem up at once: fx runs all
OnStarthooks back-to-back, thenstartAgentloads and schedules every check. That burst of concurrent allocation makes the Go heap overshoot well past its steady-state size before GC catches up, and because the runtime returns memory to the OS lazily, the peak becomes the resident high-water-mark that stays for the life of the process. Inenv:single-machine-performancethis startup spike — not steady state — is what memory regression gates measure.This adds an opt-in way to spread that startup work over time so the peak stays close to steady state, at the cost of a slightly slower ramp to full operation.
What changed
A new
comp/core/startupsequencercomponent. Instead of doing heavy work directly in itsOnStarthook, a subsystem registers it withDefer(stage, name, fn). When staged startup is enabled, the sequencer runs the deferred work in ordered stages from a background goroutine, callingruntime.GC()+debug.FreeOSMemory()between stages to return transient memory to the OS before the next stage allocates.staged_start.enabled(default false),staged_start.stage_interval(default 5s), andstaged_start.free_os_memory(default true). When disabled,Deferruns the work inline, so behavior is identical to today.core.WithStagedStartup()swaps in the real one, wired only intoagent run. The standalone DogStatsD binary andcore.MockBundle()get the no-op so their fx graphs still resolve.OnStart(ingest stage), and autodiscoveryLoadAndRun(checks stage — the single largest transient burst). Forwarder/aggregator/API stay immediate. Remaining subsystems (metadata, OTel, process agent, network device monitoring) are a one-line follow-up each.Note: fx runs
OnStarthooks sequentially, so hooks must register-and-return rather than block on a later stage — otherwise later hooks would never run. The sequencer runs the deferred work afterapp.Start()returns.Validation
New unit tests cover the sequencer's inline-when-disabled behavior, stage ordering, late registration, and context-cancellation. Existing dogstatsd-server, logs-agent, and
agent runfx-graph tests continue to pass with the new dependency wired through.