Skip to content

Commit 6358564

Browse files
committed
Merge branch 'feature/add-aimlapi-models-provider' into release
2 parents 57294c8 + 177199a commit 6358564

516 files changed

Lines changed: 26463 additions & 12730 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
---
2+
name: openclaw-test-heap-leaks
3+
description: Investigate `pnpm test` memory growth, Vitest worker OOMs, and suspicious RSS increases in OpenClaw using the `scripts/test-parallel.mjs` heap snapshot tooling. Use when Codex needs to reproduce test-lane memory growth, collect repeated `.heapsnapshot` files, compare snapshots from the same worker PID, distinguish transformed-module retention from real data leaks, and fix or reduce the impact by patching cleanup logic or isolating hotspot tests.
4+
---
5+
6+
# OpenClaw Test Heap Leaks
7+
8+
Use this skill for test-memory investigations. Do not guess from RSS alone when heap snapshots are available.
9+
10+
## Workflow
11+
12+
1. Reproduce the failing shape first.
13+
- Match the real entrypoint if possible. For Linux CI-style unit failures, start with:
14+
- `pnpm canvas:a2ui:bundle && OPENCLAW_TEST_MEMORY_TRACE=1 OPENCLAW_TEST_HEAPSNAPSHOT_INTERVAL_MS=60000 OPENCLAW_TEST_HEAPSNAPSHOT_DIR=.tmp/heapsnap OPENCLAW_TEST_WORKERS=2 OPENCLAW_TEST_MAX_OLD_SPACE_SIZE_MB=6144 pnpm test`
15+
- Keep `OPENCLAW_TEST_MEMORY_TRACE=1` enabled so the wrapper prints per-file RSS summaries alongside the snapshots.
16+
- If the report is about a specific shard or worker budget, preserve that shape.
17+
18+
2. Wait for repeated snapshots before concluding anything.
19+
- Take at least two intervals from the same lane.
20+
- Compare snapshots from the same PID inside one lane directory such as `.tmp/heapsnap/unit-fast/`.
21+
- Use `scripts/heapsnapshot-delta.mjs` to compare either two files directly or the earliest/latest pair per PID in one lane directory.
22+
23+
3. Classify the growth before choosing a fix.
24+
- If growth is dominated by Vite/Vitest transformed source strings, `Module`, `system / Context`, bytecode, descriptor arrays, or property maps, treat it as retained module graph growth in long-lived workers.
25+
- If growth is dominated by app objects, caches, buffers, server handles, timers, mock state, sqlite state, or similar runtime objects, treat it as a likely cleanup or lifecycle leak.
26+
27+
4. Fix the right layer.
28+
- For retained transformed-module growth in shared workers:
29+
- Move hotspot files out of `unit-fast` by updating `test/fixtures/test-parallel.behavior.json`.
30+
- Prefer `singletonIsolated` for files that are safe alone but inflate shared worker heaps.
31+
- If the file should already have been peeled out by timings but is absent from `test/fixtures/test-timings.unit.json`, call that out explicitly. Missing timings are a scheduling blind spot.
32+
- For real leaks:
33+
- Patch the implicated test or runtime cleanup path.
34+
- Look for missing `afterEach`/`afterAll`, module-reset gaps, retained global state, unreleased DB handles, or listeners/timers that survive the file.
35+
36+
5. Verify with the most direct proof.
37+
- Re-run the targeted lane or file with heap snapshots enabled if the suite still finishes in reasonable time.
38+
- If snapshot overhead pushes tests over Vitest timeouts, fall back to the same lane without snapshots and confirm the RSS trend or OOM is reduced.
39+
- For wrapper-only changes, at minimum verify the expected lanes start and the snapshot files are written.
40+
41+
## Heuristics
42+
43+
- Do not call everything a leak. In this repo, large `unit-fast` growth can be a worker-lifetime problem rather than an application object leak.
44+
- `scripts/test-parallel.mjs` and `scripts/test-parallel-memory.mjs` are the primary control points for wrapper diagnostics.
45+
- The lane names printed by `[test-parallel] start ...` and `[test-parallel][mem] summary ...` tell you where to focus.
46+
- When one or two files account for most of the delta and they are missing from timings, reducing impact by isolating them is usually the first pragmatic fix.
47+
- When the same retained object families grow across multiple intervals in the same worker PID, trust the snapshots over intuition.
48+
49+
## Snapshot Comparison
50+
51+
- Direct comparison:
52+
- `node .agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs before.heapsnapshot after.heapsnapshot`
53+
- Auto-select earliest/latest snapshots per PID within one lane:
54+
- `node .agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs --lane-dir .tmp/heapsnap/unit-fast`
55+
- Useful flags:
56+
- `--top 40`
57+
- `--min-kb 32`
58+
- `--pid 16133`
59+
60+
Read the top positive deltas first. Large positive growth in module-transform artifacts suggests lane isolation; large positive growth in runtime objects suggests a real leak.
61+
62+
## Output Expectations
63+
64+
When using this skill, report:
65+
66+
- The exact reproduce command.
67+
- Which lane and PID were compared.
68+
- The dominant retained object families from the snapshot delta.
69+
- Whether the issue is a real leak or shared-worker retained module growth.
70+
- The concrete fix or impact-reduction patch.
71+
- What you verified, and what snapshot overhead prevented you from verifying.
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
interface:
2+
display_name: "Test Heap Leaks"
3+
short_description: "Investigate test OOMs with heap snapshots"
4+
default_prompt: "Use $openclaw-test-heap-leaks to investigate test memory growth with heap snapshots and reduce its impact."
Lines changed: 265 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,265 @@
1+
#!/usr/bin/env node
2+
3+
import fs from "node:fs";
4+
import path from "node:path";
5+
6+
function printUsage() {
7+
console.error(
8+
"Usage: node heapsnapshot-delta.mjs <before.heapsnapshot> <after.heapsnapshot> [--top N] [--min-kb N]",
9+
);
10+
console.error(
11+
" or: node heapsnapshot-delta.mjs --lane-dir <dir> [--pid PID] [--top N] [--min-kb N]",
12+
);
13+
}
14+
15+
function fail(message) {
16+
console.error(message);
17+
process.exit(1);
18+
}
19+
20+
function parseArgs(argv) {
21+
const options = {
22+
top: 30,
23+
minKb: 64,
24+
laneDir: null,
25+
pid: null,
26+
files: [],
27+
};
28+
29+
for (let index = 0; index < argv.length; index += 1) {
30+
const arg = argv[index];
31+
if (arg === "--top") {
32+
options.top = Number.parseInt(argv[index + 1] ?? "", 10);
33+
index += 1;
34+
continue;
35+
}
36+
if (arg === "--min-kb") {
37+
options.minKb = Number.parseInt(argv[index + 1] ?? "", 10);
38+
index += 1;
39+
continue;
40+
}
41+
if (arg === "--lane-dir") {
42+
options.laneDir = argv[index + 1] ?? null;
43+
index += 1;
44+
continue;
45+
}
46+
if (arg === "--pid") {
47+
options.pid = Number.parseInt(argv[index + 1] ?? "", 10);
48+
index += 1;
49+
continue;
50+
}
51+
options.files.push(arg);
52+
}
53+
54+
if (!Number.isFinite(options.top) || options.top <= 0) {
55+
fail("--top must be a positive integer");
56+
}
57+
if (!Number.isFinite(options.minKb) || options.minKb < 0) {
58+
fail("--min-kb must be a non-negative integer");
59+
}
60+
if (options.pid !== null && (!Number.isInteger(options.pid) || options.pid <= 0)) {
61+
fail("--pid must be a positive integer");
62+
}
63+
64+
return options;
65+
}
66+
67+
function parseHeapFilename(filePath) {
68+
const base = path.basename(filePath);
69+
const match = base.match(
70+
/^Heap\.(?<stamp>\d{8}\.\d{6})\.(?<pid>\d+)\.0\.(?<seq>\d+)\.heapsnapshot$/u,
71+
);
72+
if (!match?.groups) {
73+
return null;
74+
}
75+
return {
76+
filePath,
77+
pid: Number.parseInt(match.groups.pid, 10),
78+
stamp: match.groups.stamp,
79+
sequence: Number.parseInt(match.groups.seq, 10),
80+
};
81+
}
82+
83+
function resolvePair(options) {
84+
if (options.laneDir) {
85+
const entries = fs
86+
.readdirSync(options.laneDir)
87+
.map((name) => parseHeapFilename(path.join(options.laneDir, name)))
88+
.filter((entry) => entry !== null)
89+
.filter((entry) => options.pid === null || entry.pid === options.pid)
90+
.toSorted((left, right) => {
91+
if (left.pid !== right.pid) {
92+
return left.pid - right.pid;
93+
}
94+
if (left.stamp !== right.stamp) {
95+
return left.stamp.localeCompare(right.stamp);
96+
}
97+
return left.sequence - right.sequence;
98+
});
99+
100+
if (entries.length === 0) {
101+
fail(`No matching heap snapshots found in ${options.laneDir}`);
102+
}
103+
104+
const groups = new Map();
105+
for (const entry of entries) {
106+
const group = groups.get(entry.pid) ?? [];
107+
group.push(entry);
108+
groups.set(entry.pid, group);
109+
}
110+
111+
const candidates = Array.from(groups.values())
112+
.map((group) => ({
113+
pid: group[0].pid,
114+
before: group[0],
115+
after: group.at(-1),
116+
count: group.length,
117+
}))
118+
.filter((entry) => entry.count >= 2);
119+
120+
if (candidates.length === 0) {
121+
fail(`Need at least two snapshots for one PID in ${options.laneDir}`);
122+
}
123+
124+
const chosen =
125+
options.pid !== null
126+
? (candidates.find((entry) => entry.pid === options.pid) ?? null)
127+
: candidates.toSorted((left, right) => right.count - left.count || left.pid - right.pid)[0];
128+
129+
if (!chosen) {
130+
fail(`No PID with at least two snapshots matched in ${options.laneDir}`);
131+
}
132+
133+
return {
134+
before: chosen.before.filePath,
135+
after: chosen.after.filePath,
136+
pid: chosen.pid,
137+
snapshotCount: chosen.count,
138+
};
139+
}
140+
141+
if (options.files.length !== 2) {
142+
printUsage();
143+
process.exit(1);
144+
}
145+
146+
return {
147+
before: options.files[0],
148+
after: options.files[1],
149+
pid: null,
150+
snapshotCount: 2,
151+
};
152+
}
153+
154+
function loadSummary(filePath) {
155+
const data = JSON.parse(fs.readFileSync(filePath, "utf8"));
156+
const meta = data.snapshot?.meta;
157+
if (!meta) {
158+
fail(`Invalid heap snapshot: ${filePath}`);
159+
}
160+
161+
const nodeFieldCount = meta.node_fields.length;
162+
const typeNames = meta.node_types[0];
163+
const strings = data.strings;
164+
const typeIndex = meta.node_fields.indexOf("type");
165+
const nameIndex = meta.node_fields.indexOf("name");
166+
const selfSizeIndex = meta.node_fields.indexOf("self_size");
167+
168+
const summary = new Map();
169+
for (let offset = 0; offset < data.nodes.length; offset += nodeFieldCount) {
170+
const type = typeNames[data.nodes[offset + typeIndex]];
171+
const name = strings[data.nodes[offset + nameIndex]];
172+
const selfSize = data.nodes[offset + selfSizeIndex];
173+
const key = `${type}\t${name}`;
174+
const current = summary.get(key) ?? {
175+
type,
176+
name,
177+
selfSize: 0,
178+
count: 0,
179+
};
180+
current.selfSize += selfSize;
181+
current.count += 1;
182+
summary.set(key, current);
183+
}
184+
return {
185+
nodeCount: data.snapshot.node_count,
186+
summary,
187+
};
188+
}
189+
190+
function formatBytes(bytes) {
191+
if (Math.abs(bytes) >= 1024 ** 2) {
192+
return `${(bytes / 1024 ** 2).toFixed(2)} MiB`;
193+
}
194+
if (Math.abs(bytes) >= 1024) {
195+
return `${(bytes / 1024).toFixed(1)} KiB`;
196+
}
197+
return `${bytes} B`;
198+
}
199+
200+
function formatDelta(bytes) {
201+
return `${bytes >= 0 ? "+" : "-"}${formatBytes(Math.abs(bytes))}`;
202+
}
203+
204+
function truncate(text, maxLength) {
205+
return text.length <= maxLength ? text : `${text.slice(0, maxLength - 1)}…`;
206+
}
207+
208+
function main() {
209+
const options = parseArgs(process.argv.slice(2));
210+
const pair = resolvePair(options);
211+
const before = loadSummary(pair.before);
212+
const after = loadSummary(pair.after);
213+
const minBytes = options.minKb * 1024;
214+
215+
const rows = [];
216+
for (const [key, next] of after.summary) {
217+
const previous = before.summary.get(key) ?? { selfSize: 0, count: 0 };
218+
const sizeDelta = next.selfSize - previous.selfSize;
219+
const countDelta = next.count - previous.count;
220+
if (sizeDelta < minBytes) {
221+
continue;
222+
}
223+
rows.push({
224+
type: next.type,
225+
name: next.name,
226+
sizeDelta,
227+
countDelta,
228+
afterSize: next.selfSize,
229+
afterCount: next.count,
230+
});
231+
}
232+
233+
rows.sort(
234+
(left, right) => right.sizeDelta - left.sizeDelta || right.countDelta - left.countDelta,
235+
);
236+
237+
console.log(`before: ${pair.before}`);
238+
console.log(`after: ${pair.after}`);
239+
if (pair.pid !== null) {
240+
console.log(`pid: ${pair.pid} (${pair.snapshotCount} snapshots found)`);
241+
}
242+
console.log(
243+
`nodes: ${before.nodeCount} -> ${after.nodeCount} (${after.nodeCount - before.nodeCount >= 0 ? "+" : ""}${after.nodeCount - before.nodeCount})`,
244+
);
245+
console.log(`filter: top=${options.top} min=${options.minKb} KiB`);
246+
console.log("");
247+
248+
if (rows.length === 0) {
249+
console.log("No entries exceeded the minimum delta.");
250+
return;
251+
}
252+
253+
for (const row of rows.slice(0, options.top)) {
254+
console.log(
255+
[
256+
formatDelta(row.sizeDelta).padStart(11),
257+
`count ${row.countDelta >= 0 ? "+" : ""}${row.countDelta}`.padStart(10),
258+
row.type.padEnd(16),
259+
truncate(row.name || "(empty)", 96),
260+
].join(" "),
261+
);
262+
}
263+
}
264+
265+
main();

.dockerignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
.git
22
.worktrees
33

4-
# Sensitive files – docker-setup.sh writes .env with OPENCLAW_GATEWAY_TOKEN
4+
# Sensitive files – scripts/docker/setup.sh writes .env with OPENCLAW_GATEWAY_TOKEN
55
# into the project root; keep it out of the build context.
66
.env
77
.env.*

.github/labeler.yml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,7 +165,10 @@
165165
- "Dockerfile.*"
166166
- "docker-compose.yml"
167167
- "docker-setup.sh"
168+
- "setup-podman.sh"
168169
- ".dockerignore"
170+
- "scripts/docker/setup.sh"
171+
- "scripts/podman/setup.sh"
169172
- "scripts/**/*docker*"
170173
- "scripts/**/Dockerfile*"
171174
- "scripts/sandbox-*.sh"
@@ -290,6 +293,10 @@
290293
- changed-files:
291294
- any-glob-to-any-file:
292295
- "extensions/synthetic/**"
296+
"extensions: tavily":
297+
- changed-files:
298+
- any-glob-to-any-file:
299+
- "extensions/tavily/**"
293300
"extensions: talk-voice":
294301
- changed-files:
295302
- any-glob-to-any-file:

0 commit comments

Comments
 (0)