Conversation
Fuse memray-deep allocation tracing with the Dask task graph and stream it live. - deepmem: memray driven as a library, epoch-rotated, folded to the first user source line; captures per-line high-water marks and full call stacks (schema v6). Opt-in via deep=True on register()/LocalProfiler; degrades to Tier-1 sampling where memray isn't importable. Never crashes or OOMs the job. - Worker plugin emits live WorkerStatus heartbeats + deep epochs; scheduler plugin death events joined with chunk metadata and allocation lines. - Collector: TimescaleDB backend (hypertables) as default behind a StoreProtocol, SQLite kept for tests/dev; asyncio pub/sub hub + /ws WebSocket fan-out; per-line, per-task, per-layer-timeline, flamegraph and worker endpoints. - Runs record origin hostname + IP so a shared collector is a team hub. - Source attribution reads the full multi-line statement, not just one line.
Next.js dashboard streaming over WebSocket, in Dask's warm palette. - Live store (WebSocket + REST seed) with a collapsible runs sidebar that updates in real time; activity-based live indicator. - Workers table, global + per-worker Task stream (canvas, zoom/pan/box-zoom), whole-graph canvas DAG with deep zoom + edge highlighting. - Memory: canvas memory-over-time with click-to-inspect spike explorer, per-layer allocations-over-time, a real per-worker icicle flamegraph (memray tree read), and the peak-by-line / peak-by-task tables. - Dedicated Timeline page, in-app delete modal, SVG logo + favicon.
- examples covering the breadth of Dask: distributed/deep OOM demos, a minutes-long big pipeline, a self-limiting crash, and one per collection type (dask.delayed, dask.dataframe, dask.bag, xarray on Zarr and NetCDF). - Packaging: PyPI metadata (authors, urls, classifiers), MIT LICENSE. - CI workflow: ruff + mypy + pytest + package build + web build. - README rewritten; examples/README expanded.
- .github/workflows/workflow-pypi.yml: on a v* tag, build + publish to PyPI via Trusted Publishing (OIDC, no token), push collector + dashboard images to GHCR, and create the GitHub release. - CONTRIBUTING.md: two-trunk model, Conventional Commits, uv/ruff/mypy/pytest, release process. - CHANGELOG.md (Keep a Changelog) and daskgenie.__version__. - .pre-commit-config.yaml running ruff + mypy via uv.
* docs(readme): absolute logo URL for PyPI + status badges Serve the logo from jsDelivr (absolute https) so it renders on the PyPI project page, not just on GitHub. Add PyPI version, Python versions, CI, and license badges. * docs(readme): point logo + license links at develop (file not on main yet)
* docs: MkDocs Material site + gh-pages deploy - mkdocs.yml (Material theme, Dask-orange palette) with mkdocstrings API reference, snippet-included CONTRIBUTING and CHANGELOG. - docs/ pages: install, quick start, dashboard, deep memory, local schedulers, configuration, examples, API. - .github/workflows/docs.yml deploys to gh-pages on push to develop. - docs dependency group (mkdocs-material, mkdocstrings). * ci(docs): deploy from main only, not develop
polymood
added a commit
that referenced
this pull request
Jul 3, 2026
* feat: deep memory engine (memray) + TimescaleDB + realtime WebSocket Fuse memray-deep allocation tracing with the Dask task graph and stream it live. - deepmem: memray driven as a library, epoch-rotated, folded to the first user source line; captures per-line high-water marks and full call stacks (schema v6). Opt-in via deep=True on register()/LocalProfiler; degrades to Tier-1 sampling where memray isn't importable. Never crashes or OOMs the job. - Worker plugin emits live WorkerStatus heartbeats + deep epochs; scheduler plugin death events joined with chunk metadata and allocation lines. - Collector: TimescaleDB backend (hypertables) as default behind a StoreProtocol, SQLite kept for tests/dev; asyncio pub/sub hub + /ws WebSocket fan-out; per-line, per-task, per-layer-timeline, flamegraph and worker endpoints. - Runs record origin hostname + IP so a shared collector is a team hub. - Source attribution reads the full multi-line statement, not just one line. * feat: real-time dashboard rebuild Next.js dashboard streaming over WebSocket, in Dask's warm palette. - Live store (WebSocket + REST seed) with a collapsible runs sidebar that updates in real time; activity-based live indicator. - Workers table, global + per-worker Task stream (canvas, zoom/pan/box-zoom), whole-graph canvas DAG with deep zoom + edge highlighting. - Memory: canvas memory-over-time with click-to-inspect spike explorer, per-layer allocations-over-time, a real per-worker icicle flamegraph (memray tree read), and the peak-by-line / peak-by-task tables. - Dedicated Timeline page, in-app delete modal, SVG logo + favicon. * feat: examples across Dask + packaging, docs, CI - examples covering the breadth of Dask: distributed/deep OOM demos, a minutes-long big pipeline, a self-limiting crash, and one per collection type (dask.delayed, dask.dataframe, dask.bag, xarray on Zarr and NetCDF). - Packaging: PyPI metadata (authors, urls, classifiers), MIT LICENSE. - CI workflow: ruff + mypy + pytest + package build + web build. - README rewritten; examples/README expanded. * chore: release workflow, contributing guide, changelog, pre-commit - .github/workflows/workflow-pypi.yml: on a v* tag, build + publish to PyPI via Trusted Publishing (OIDC, no token), push collector + dashboard images to GHCR, and create the GitHub release. - CONTRIBUTING.md: two-trunk model, Conventional Commits, uv/ruff/mypy/pytest, release process. - CHANGELOG.md (Keep a Changelog) and daskgenie.__version__. - .pre-commit-config.yaml running ruff + mypy via uv. * docs: PyPI-ready logo + README badges (#2) * docs(readme): absolute logo URL for PyPI + status badges Serve the logo from jsDelivr (absolute https) so it renders on the PyPI project page, not just on GitHub. Add PyPI version, Python versions, CI, and license badges. * docs(readme): point logo + license links at develop (file not on main yet) * docs: MkDocs site + gh-pages (#3) * docs: MkDocs Material site + gh-pages deploy - mkdocs.yml (Material theme, Dask-orange palette) with mkdocstrings API reference, snippet-included CONTRIBUTING and CHANGELOG. - docs/ pages: install, quick start, dashboard, deep memory, local schedulers, configuration, examples, API. - .github/workflows/docs.yml deploys to gh-pages on push to develop. - docs dependency group (mkdocs-material, mkdocstrings). * ci(docs): deploy from main only, not develop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.