Skip to content
86 changes: 86 additions & 0 deletions bake/async/service/supervisor.rb
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,98 @@ def status
end
end

# List the connection IDs of all registered workers.
def workers
client do |connection|
supervisor = connection[:supervisor]
supervisor.keys
end
end

# Dump the object space of a worker to a file on the worker's filesystem.
#
# @parameter connection_id [Integer] The connection ID of the worker to target.
# @parameter path [String] The file path where the worker should write the dump.
# @parameter shapes [Boolean] Whether to include Ruby shape-tree records.
def memory_dump(connection_id:, path:, shapes: true)
with_worker(connection_id) do |worker|
worker.memory_dump(path: path, shapes: shapes)
end
end

# Start recording object allocation metadata in a worker.
#
# @parameter connection_id [Integer] The connection ID of the worker to target.
def allocation_trace_start(connection_id:)
with_worker(connection_id) do |worker|
worker.allocation_trace_start
end
end

# Stop recording object allocations, dump the traced heap, and clear the metadata.
#
# @parameter connection_id [Integer] The connection ID of the worker to target.
# @parameter path [String] The file path where the worker should write the dump.
# @parameter shapes [Boolean] Whether to include Ruby shape-tree records.
def allocation_trace_stop(connection_id:, path:, shapes: true)
with_worker(connection_id) do |worker|
worker.allocation_trace_stop(path: path, shapes: shapes)
end
end

# Dump the fiber scheduler hierarchy of a worker.
#
# @parameter connection_id [Integer] The connection ID of the worker to target.
# @parameter path [String | Nil] An optional file path on the worker's filesystem.
# @parameter log [String | Nil] An optional message to log with the dump.
def scheduler_dump(connection_id:, path: nil, log: nil)
with_worker(connection_id) do |worker|
worker.scheduler_dump(path: path, log: log)
end
end

# Dump information about all threads in a worker.
#
# @parameter connection_id [Integer] The connection ID of the worker to target.
# @parameter path [String | Nil] An optional file path on the worker's filesystem.
def thread_dump(connection_id:, path: nil)
with_worker(connection_id) do |worker|
worker.thread_dump(path: path)
end
end

# Start garbage collection profiling in a worker.
#
# @parameter connection_id [Integer] The connection ID of the worker to target.
def garbage_profile_start(connection_id:)
with_worker(connection_id) do |worker|
worker.garbage_profile_start
end
end

# Stop garbage collection profiling in a worker and return or save the results.
#
# @parameter connection_id [Integer] The connection ID of the worker to target.
# @parameter path [String | Nil] An optional file path on the worker's filesystem.
def garbage_profile_stop(connection_id:, path: nil)
with_worker(connection_id) do |worker|
worker.garbage_profile_stop(path: path)
end
end

private

def endpoint
Async::Service::Supervisor.endpoint
end

def with_worker(connection_id)
client do |connection|
supervisor = connection[:supervisor]
yield supervisor[connection_id]
end
end

def client(&block)
Sync do
Async::Service::Supervisor::Client.new(endpoint: self.endpoint).connect(&block)
Expand Down
36 changes: 10 additions & 26 deletions context/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,44 +149,28 @@ end
The supervisor can collect various diagnostics from workers on demand:

- **Memory dumps**: Full heap dumps for memory analysis via `ObjectSpace.dump_all`.
- **Memory samples**: Lightweight sampling to identify memory leaks.
- **Thread dumps**: Stack traces of all threads.
- **Scheduler dumps**: Async fiber hierarchy
- **Garbage collection profiles**: GC performance data
- **Scheduler dumps**: Async fiber hierarchy.
- **Garbage collection profiles**: GC performance data.

These can be triggered programmatically or via command-line tools (when available).
These can be triggered programmatically or with Bake tasks.

#### Memory Leak Diagnosis

To identify memory leaks, you can use the memory sampling feature which is much lighter weight than a full memory dump. It tracks allocations over a time period and focuses on retained objects.

**Using the bake task:**
Start by listing the workers registered with the supervisor:

```bash
# Sample for 30 seconds and print report to console
$ bake async:container:supervisor:memory_sample duration=30
$ bake async:service:supervisor:workers
```

**Programmatically:**
Then capture heap dumps from the same worker before and after the suspected growth period:

```ruby
# Assuming you have a connection to a worker:
result = connection.call(do: :memory_sample, duration: 30)
puts result[:data]
```bash
$ bake async:service:supervisor:memory_dump connection_id=1 path=/var/tmp/worker-before.json
$ bake async:service:supervisor:memory_dump connection_id=1 path=/var/tmp/worker-after.json
```

This will sample memory allocations for the specified duration, then force a garbage collection and return a JSON report showing what objects were allocated during that period and retained after GC. Late-lifecycle allocations that are retained are likely memory leaks.

The JSON report includes:
- `total_allocated`: Total allocated memory and count
- `total_retained`: Total retained memory and count
- `by_gem`: Breakdown by gem/library
- `by_file`: Breakdown by source file
- `by_location`: Breakdown by specific file:line locations
- `by_class`: Breakdown by object class
- `strings`: String allocation analysis

This is much more efficient than `do: :memory_dump` which uses `ObjectSpace.dump_all` and can be slow and blocking on large heaps. The JSON format also makes it easy to integrate with monitoring and analysis tools.
Heap dumps are heavyweight and may contain sensitive application data. See the [Memory Diagnostics](../memory-diagnostics/index) guide for the complete capture, comparison, and GC profiling workflow.

## Advanced Usage

Expand Down
4 changes: 4 additions & 0 deletions context/index.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@ files:
title: Memory Monitor
description: This guide explains how to use the <code class="language-ruby">Async::Service::Supervisor::MemoryMonitor</code>
to detect and restart workers that exceed memory limits or develop memory leaks.
- path: memory-diagnostics.md
title: Memory Diagnostics
description: This guide explains how to capture Ruby heap dumps and garbage collection
profiles from supervised workers, then use them to investigate memory growth.
- path: process-monitor.md
title: Process Monitor
description: This guide explains how to use the <code class="language-ruby">Async::Service::Supervisor::ProcessMonitor</code>
Expand Down
Loading
Loading