Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 21 additions & 10 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,24 +74,35 @@ Event classes:
- `freeze_recovered`
- `freeze_stack` (optional; when `captureStack` is enabled)

### 4) Stack capture (opt-in)
### 4) Stack / profile capture (opt-in)

When `captureStack` is enabled, the monitor thread calls V8 `RequestInterrupt`.
**Interrupt mode** (`mode: "interrupt"`, including `captureStack: true`):

When enabled, the monitor thread calls V8 `RequestInterrupt`.
The interrupt callback runs on the isolate thread, captures `v8::StackTrace`,
and only stashes frames (+ queues a pending event). The monitor thread then
writes `freeze_stack` / notifies JS — never logger or N-API from the interrupt.

- Default sampling: on `freeze_started` and each `freeze_heartbeat` (`on: "both"`).
- `"started"` / `"heartbeat"` narrow when interrupts are requested.
- Unique stack shapes are aggregated per freeze (capped by `maxSamples`); on
`freeze_recovered`, `stack` is the most frequent sample and `stack_samples`
lists `{ count, stack }` sorted by count descending.
- A single interrupt sample is not reliable attribution under many concurrent
async handlers (often only `processTicksAndRejections`); multi-sample helps
longer freezes but is not a CPU profile. A future experimental profiler mode
may cover that case.
- Sync I/O / native blocks may never reach a safepoint → `stack_status: "unavailable"` (no `stack` field).
- `freeze_stack` reuses `rss_mb` / `cpu_pct` from the latest lifecycle event so a near-zero-delta CPU sample is not emitted.
async handlers (often only `processTicksAndRejections`); the sample line is
the safepoint after native work, not always the hottest statement.

**Profile mode** (`mode: "profile"`):

- Arms V8 `CpuProfiler` when lag ≥ `freezeThresholdMs / 2`; discards if lag
drops without a freeze; on recover stops and attaches top hit-count paths as
`stack_samples` with `stack_mode: "profile"` (no live `freeze_stack` events).
- Start/stop/dispose run only on the isolate thread via `RequestInterrupt`.
- Adds sampling overhead while armed; still experimental / ABI-gated.

Shared:

- Sync I/O / native blocks may yield `stack_status: "unavailable"`.
- `freeze_stack` reuses `rss_mb` / `cpu_pct` from the latest lifecycle event.
- Implemented in-core (experimental); not a separate package.

## Freeze Detection Model
Expand Down Expand Up @@ -142,9 +153,9 @@ Recommended fields:
Optional (experimental, when `captureStack` is enabled):

- `stack_status` — `"ok"` \| `"unavailable"`
- `stack_mode` — `"interrupt"`
- `stack_mode` — `"interrupt"` \| `"profile"`
- `stack` — string frames; present only when `stack_status` is `"ok"`
(on recovered: most frequent sample)
(on recovered: most frequent interrupt sample or hottest profile path)
- `stack_samples` — on recovered: `[{ count, stack }, ...]` sorted by count descending

## Error Handling and Safety
Expand Down
16 changes: 10 additions & 6 deletions COMPATIBILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,19 +42,23 @@ Notes on `ts`:
- Opt-in stack capture is experimental and may change shape without a major bump until marked stable:
- config: `captureStack` (`false` \| `true` \| `{ mode, on, maxFrames, maxSamples }`); default `false`;
- `true` expands to `{ mode: "interrupt", on: "both", maxFrames: 50, maxSamples: 8 }`;
- additive event value `freeze_stack` (same channels as other freeze events);
- payload fields: `stack_status`, `stack_mode`, `stack` (`stack` only when status is `"ok"`);
- `mode`: `"interrupt"` (RequestInterrupt samples) or `"profile"` (V8 CpuProfiler;
early-arm at `freezeThresholdMs / 2`; `on` ignored);
- additive event value `freeze_stack` (interrupt mode; same channels as other freeze events);
- payload fields: `stack_status`, `stack_mode` (`"interrupt"` \| `"profile"`),
`stack` (`stack` only when status is `"ok"`);
- on `freeze_recovered`: optional `stack_samples` (`[{ count, stack }, ...]`, count-desc)
when at least one interrupt sample succeeded; `stack` is the most frequent sample;
- uses V8 `RequestInterrupt` (JS busy-loop stacks; sync I/O / native blocks may yield `unavailable`;
under concurrent async work a sample may show only the promise microtask runner);
when at least one sample succeeded; `stack` is the top sample (most frequent or hottest);
- interrupt mode: JS busy-loop stacks; sync I/O / native blocks may yield `unavailable`;
under concurrent async work a sample may show only the promise microtask runner;
- profile mode: hit counts while armed; thin/`unavailable` possible if start never ran
before recover; sampling overhead while armed;
- stack frames may include absolute file paths — treat logs as sensitive when enabled;
- if the loaded native addon ABI (`NODE_MODULE_VERSION`) differs from the
runtime (no matching published prebuild for this Node major),
`captureStack` is disabled with a warning.
- Linux prebuilds are libc-tagged (`glibc` / `musl`); loading an untagged
glibc binary on Alpine is unsupported and may SIGSEGV under `captureStack`.
- A future experimental CPU-profile capture mode is reserved and not part of v1 yet.

## Behavioral guarantees

Expand Down
22 changes: 12 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,24 +104,24 @@ node examples/log-max-bytes.js

### `captureStack` (experimental)

Uses V8 `RequestInterrupt` to sample the JS stack when a freeze is detected. Works best for JS busy-loops; sync I/O / native blocks may leave `stack_status: "unavailable"`.
Two modes:

Under heavy parallel async work, a single sample often lands in `processTicksAndRejections` only — that does **not** identify which of many concurrent handlers blocked the loop. Prefer `on: "both"` (the `true` default) so longer freezes re-sample on heartbeat; on `freeze_recovered`, `stack` is the **most frequent** shape and `stack_samples` lists unique stacks with counts. Nearby request-completion logs are weak attribution under concurrency.
**`interrupt` (default via `true`)** — V8 `RequestInterrupt` snapshots. Works best for JS busy-loops; sync I/O / native blocks may leave `stack_status: "unavailable"`. Under heavy parallel async work, a sample often lands in `processTicksAndRejections` only. Prefer `on: "both"` so longer freezes re-sample; on recovered, `stack` is the most frequent shape and `stack_samples` lists counts. Interrupt stacks show the JS line at the next safepoint — often the statement **after** a long native call (e.g. `JSON.parse`).

Stack capture calls V8 C++ APIs. Release CI ships one ABI-tagged binary per supported Node major; `node-gyp-build` picks the match at install/load time.
**`profile`** — V8 `CpuProfiler` over the stall. Arms when lag ≥ `freezeThresholdMs / 2`, stops on recover (or discards if lag drops without freezing). On recovered, `stack_mode: "profile"` and `stack_samples` are top hot paths by hit count (no live `freeze_stack` spam). Better attribution for CPU-bound work that yields to safepoints while profiling; a single unbroken native call that never yields before recover may still produce a thin/`unavailable` profile. Adds sampling overhead while armed — keep opt-in.

A future experimental CPU-profile mode (V8 `CpuProfiler` over the freeze window) is planned for stronger attribution; not available yet.
Stack/profile capture uses V8 C++ APIs. Release CI ships one ABI-tagged binary per supported Node major; `node-gyp-build` picks the match at install/load time.

| Value | Meaning |
| --- | --- |
| `false` / omit | disabled (default) |
| `true` | `{ mode: "interrupt", on: "both", maxFrames: 50, maxSamples: 8 }` |
| `{ mode, on, maxFrames, maxSamples }` | `mode`: `"interrupt"` only; `on`: `"started"` \| `"heartbeat"` \| `"both"`; `maxFrames`: `1..256`; `maxSamples`: `1..32` unique shapes retained per freeze |
| `{ mode, on, maxFrames, maxSamples }` | `mode`: `"interrupt"` \| `"profile"`; `on` (interrupt only): `"started"` \| `"heartbeat"` \| `"both"`; `maxFrames`: `1..256`; `maxSamples`: `1..32` |

When enabled, native logs / JS events may include:

- `freeze_stack` — live sample (`channel: "freeze"`); `rss_mb` / `cpu_pct` are copied from the latest lifecycle event (started/heartbeat), not re-sampled
- on `freeze_recovered`: `stack_status`, `stack_mode`, and `stack` only when status is `"ok"`; `stack_samples` (`[{ count, stack }, ...]`, count-desc) when at least one sample succeeded
- `freeze_stack` — interrupt mode live sample (`channel: "freeze"`); `rss_mb` / `cpu_pct` copied from the latest lifecycle event
- on `freeze_recovered`: `stack_status`, `stack_mode` (`"interrupt"` \| `"profile"`), and `stack` when status is `"ok"`; `stack_samples` (`[{ count, stack }, ...]`, count-desc) when samples exist
- frames often contain absolute paths — keep logs access-controlled when capture is on

### Event payload
Expand All @@ -143,7 +143,7 @@ When enabled, native logs / JS events may include:
cpu_pct: 79.92, // -1 if unavailable
// only when captureStack is enabled (on freeze_stack / freeze_recovered):
// stack_status: "ok" | "unavailable",
// stack_mode: "interrupt",
// stack_mode: "interrupt" | "profile",
// stack: ["at busyWait (test.js:12:5)", ...], // omitted when unavailable
// on freeze_recovered when samples exist:
// stack_samples: [{ count: 3, stack: ["at busyWait ..."] }, ...],
Expand All @@ -160,8 +160,10 @@ When enabled, native logs / JS events may include:
| Config throws `TypeError` / `RangeError` | Invalid options | See config table; values must be plain object + ranges |
| Log file missing | Unwritable path / missing directories | Logger fails open quietly; stderr/`both` still work; create parent dirs if you need a file |
| High `cpu_pct` during freeze | Busy-loop / CPU-bound block | Expected for sync CPU spins; use with RSS/duration context |
| No `freeze_stack` / `stack_status: "unavailable"` | Sync I/O, native addon, or interrupt never reached a V8 safepoint | Expected for non-JS blocks; check native logs around recovery; try `on: "both"` for retries |
| `stack_status: "ok"` but only `processTicksAndRejections` / `task_queues` | Interrupt landed in the promise microtask runner under async load | Expected for many concurrent awaits; use `stack_samples` + duration/RSS/CPU; do not trust nearby API method logs for attribution; raise `freezeThresholdMs` if short stalls are noise |
| No `freeze_stack` / `stack_status: "unavailable"` | Sync I/O, native addon, or interrupt never reached a V8 safepoint | Expected for non-JS blocks; check native logs around recovery; try `on: "both"` or `mode: "profile"` |
| `stack_status: "ok"` but only `processTicksAndRejections` / `task_queues` | Interrupt landed in the promise microtask runner under async load | Expected for many concurrent awaits; try `mode: "profile"`; use duration/RSS/CPU; raise `freezeThresholdMs` if short stalls are noise |
| Interrupt stack line is after `JSON.parse` / other native | Interrupt runs at the next safepoint after native returns | Expected; use `mode: "profile"` for hit-count attribution when the profiler was armed in time |
| Profile `unavailable` / empty on short native-only stalls | Profiler start interrupt could not run until after the block | Expected for one unbroken native call; lengthen work or accept interrupt function-level hint |
| `captureStack` off + ABI warning | No prebuild for this Node major / wrong binary loaded | Use Node 22/24/26, or upgrade `@js-ak/watchdog` once that ABI is published |
| `npm ci` in Debian slim tries to compile / needs Python | Linux prebuild needs newer `libstdc++` than the image, so load fails and install falls back to `node-gyp` | Use a newer base image, or upgrade `@js-ak/watchdog` (Ubuntu 22.04 prebuilds) |
| Container exit **139** / SIGSEGV (often with `captureStack`) on Alpine | glibc Linux prebuild loaded on musl | Use a release with libc-tagged + musl prebuilds; or switch to a glibc image (`node:*-bookworm-slim`); or rebuild from source on Alpine after removing `node_modules/@js-ak/watchdog/prebuilds` |
Expand Down
16 changes: 16 additions & 0 deletions native/addon.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,22 @@ bool ReadConfig(napi_env env, napi_value object,
config->capture_stack = true;

napi_value field;
if (napi_get_named_property(env, value, "mode", &field) == napi_ok) {
size_t len = 0;
napi_get_value_string_utf8(env, field, nullptr, 0, &len);
std::string mode(len, '\0');
if (napi_get_value_string_utf8(env, field, mode.data(), len + 1,
&len) == napi_ok) {
if (mode == "profile") {
config->capture_stack_mode =
jsak::watchdog::StackCaptureMode::Profile;
} else if (mode == "interrupt") {
config->capture_stack_mode =
jsak::watchdog::StackCaptureMode::Interrupt;
}
}
}

if (napi_get_named_property(env, value, "on", &field) == napi_ok) {
size_t len = 0;
napi_get_value_string_utf8(env, field, nullptr, 0, &len);
Expand Down
159 changes: 159 additions & 0 deletions native/cpu_profile.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
#pragma once

#include <algorithm>
#include <string>
#include <vector>

#include <v8-profiler.h>
#include <v8.h>

#include "watchdog.h"

namespace jsak {
namespace watchdog {
namespace {

inline constexpr const char kCpuProfileTitle[] = "js-ak/watchdog";

inline std::string FormatProfileFrame(v8::Isolate* isolate,
const v8::CpuProfileNode* node) {
if (node == nullptr) {
return "at <unknown>";
}

v8::String::Utf8Value fn(isolate, node->GetFunctionName());
v8::String::Utf8Value script(isolate, node->GetScriptResourceName());
const int line = node->GetLineNumber();
const int column = node->GetColumnNumber();

std::string out = "at ";
if (fn.length() > 0) {
out.append(*fn, static_cast<size_t>(fn.length()));
out += " (";
}

if (script.length() > 0) {
out.append(*script, static_cast<size_t>(script.length()));
} else {
out += "<anonymous>";
}
if (line > 0) {
out += ':';
out += std::to_string(line);
if (column > 0) {
out += ':';
out += std::to_string(column);
}
}

if (fn.length() > 0) {
out += ')';
}
return out;
}

inline void WalkProfileNode(v8::Isolate* isolate,
const v8::CpuProfileNode* node,
std::vector<std::string>& path,
std::vector<StackSample>& samples,
int max_frames) {
if (node == nullptr) {
return;
}

path.push_back(FormatProfileFrame(isolate, node));

const unsigned hits = node->GetHitCount();
if (hits > 0 && path.size() > 1) {
// Skip pure root-only hits; keep paths with real frames.
StackSample sample;
sample.count = hits;
// Leaf-first order to match interrupt / Error.stack style.
const size_t n = path.size();
const size_t take =
max_frames > 0 ? std::min(n, static_cast<size_t>(max_frames)) : n;
sample.stack.reserve(take);
for (size_t i = 0; i < take; i += 1) {
sample.stack.push_back(path[n - 1 - i]);
}
samples.push_back(std::move(sample));
}

const int children = node->GetChildrenCount();
for (int i = 0; i < children; i += 1) {
WalkProfileNode(isolate, node->GetChild(i), path, samples, max_frames);
}

path.pop_back();
}

} // namespace

inline std::vector<StackSample> CollectCpuProfileSamples(
v8::Isolate* isolate, const v8::CpuProfile* profile, int max_frames,
uint32_t max_samples) {
std::vector<StackSample> samples;
if (isolate == nullptr || profile == nullptr) {
return samples;
}

std::vector<std::string> path;
WalkProfileNode(isolate, profile->GetTopDownRoot(), path, samples,
max_frames);

std::stable_sort(samples.begin(), samples.end(),
[](const StackSample& a, const StackSample& b) {
return a.count > b.count;
});

const uint32_t cap = ClampStackSamples(max_samples);
if (samples.size() > static_cast<size_t>(cap)) {
samples.resize(static_cast<size_t>(cap));
}
return samples;
}

inline bool StartCpuProfiling(v8::Isolate* isolate,
v8::CpuProfiler** profiler) {
if (isolate == nullptr || profiler == nullptr) {
return false;
}
if (*profiler == nullptr) {
*profiler = v8::CpuProfiler::New(isolate);
}
if (*profiler == nullptr) {
return false;
}

v8::HandleScope handle_scope(isolate);
v8::Local<v8::String> title =
v8::String::NewFromUtf8(isolate, kCpuProfileTitle,
v8::NewStringType::kNormal)
.ToLocalChecked();
(*profiler)->StartProfiling(title, true);
return true;
}

inline v8::CpuProfile* StopCpuProfiling(v8::Isolate* isolate,
v8::CpuProfiler* profiler) {
if (isolate == nullptr || profiler == nullptr) {
return nullptr;
}
v8::HandleScope handle_scope(isolate);
v8::Local<v8::String> title =
v8::String::NewFromUtf8(isolate, kCpuProfileTitle,
v8::NewStringType::kNormal)
.ToLocalChecked();
return profiler->StopProfiling(title);
}

inline void DisposeCpuProfiler(v8::CpuProfiler** profiler) {
if (profiler == nullptr || *profiler == nullptr) {
return;
}
(*profiler)->Dispose();
*profiler = nullptr;
}

} // namespace watchdog
} // namespace jsak
Loading