Deep memory profiling + real-time dashboard#1
Closed
polymood wants to merge 4 commits into
Closed
Conversation
Fuse memray-deep allocation tracing with the Dask task graph and stream it live. - deepmem: memray driven as a library, epoch-rotated, folded to the first user source line; captures per-line high-water marks and full call stacks (schema v6). Opt-in via deep=True on register()/LocalProfiler; degrades to Tier-1 sampling where memray isn't importable. Never crashes or OOMs the job. - Worker plugin emits live WorkerStatus heartbeats + deep epochs; scheduler plugin death events joined with chunk metadata and allocation lines. - Collector: TimescaleDB backend (hypertables) as default behind a StoreProtocol, SQLite kept for tests/dev; asyncio pub/sub hub + /ws WebSocket fan-out; per-line, per-task, per-layer-timeline, flamegraph and worker endpoints. - Runs record origin hostname + IP so a shared collector is a team hub. - Source attribution reads the full multi-line statement, not just one line.
Next.js dashboard streaming over WebSocket, in Dask's warm palette. - Live store (WebSocket + REST seed) with a collapsible runs sidebar that updates in real time; activity-based live indicator. - Workers table, global + per-worker Task stream (canvas, zoom/pan/box-zoom), whole-graph canvas DAG with deep zoom + edge highlighting. - Memory: canvas memory-over-time with click-to-inspect spike explorer, per-layer allocations-over-time, a real per-worker icicle flamegraph (memray tree read), and the peak-by-line / peak-by-task tables. - Dedicated Timeline page, in-app delete modal, SVG logo + favicon.
- examples covering the breadth of Dask: distributed/deep OOM demos, a minutes-long big pipeline, a self-limiting crash, and one per collection type (dask.delayed, dask.dataframe, dask.bag, xarray on Zarr and NetCDF). - Packaging: PyPI metadata (authors, urls, classifiers), MIT LICENSE. - CI workflow: ruff + mypy + pytest + package build + web build. - README rewritten; examples/README expanded.
- .github/workflows/workflow-pypi.yml: on a v* tag, build + publish to PyPI via Trusted Publishing (OIDC, no token), push collector + dashboard images to GHCR, and create the GitHub release. - CONTRIBUTING.md: two-trunk model, Conventional Commits, uv/ruff/mypy/pytest, release process. - CHANGELOG.md (Keep a Changelog) and daskgenie.__version__. - .pre-commit-config.yaml running ruff + mypy via uv.
polymood
force-pushed
the
feat/deep-memory-realtime
branch
from
July 3, 2026 14:59
ec521eb to
6aad1ab
Compare
Owner
Author
|
Work already merged into main; switching to the develop-based two-trunk flow. Superseded. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns DaskGenie into a memray-deep, real-time Dask memory profiler and team dashboard.
Highlights
deep=True; degrades safely where memray isn't available. Schema v6.StoreProtocol, SQLite kept for tests/dev. Prometheus/metricsretained.Verification
ruff+ruff format+mypyclean; unit + deep-distributed integration tests pass.uv buildproduces a clean wheel + sdist; web builds clean.Notes / limitations
dask.array/dask.delayed; fordask.dataframe(dask-expr) and xarray the heavy work is in library C code, so allocations fold to framework frames (graph/memory/flamegraph still work).🤖 Generated with Claude Code