chore(core): implement ZSTD dict based compression by romange · Pull Request #6967 · dragonflydb/dragonfly

romange · 2026-03-24T19:00:10Z

Move ZstdDictState from per-QList unique_ptr to a thread-local raw
pointer shared across all QList instances on the same thread.
Motivation: ZSTD_CDict and ZSTD_DDict take 200-300KB each and holding them
per list kills the main motivation to save memory.

Rename AllowCompression -> AllowLZFCompression
Split TrainAndCompressZstdDict into TrainZstdDict,
CompressWithZstdDict, and CompressNodeWithDict
Add bit fields for tracking compression state:
dict_learning_failed, dict_compress_failed, dict_bulk_finished
CoolOff ZSTD branch: 3-state machine
(train -> bulk compress -> per-node compress)
Add ShutdownThread() to free thread-local dict on shard teardown
Wire list_experimental_zstd_dict_threshold flag in list_family
Add QListZstdTest suite with 6 tests

Test: ./dfly_bench -n 80000 -p 6379 -qps=0 -d 64 --key_maximum=10 --command="lpush __key__ foooooooooobbbbbbbbbbbbbbaaaaaaaaaaaaaaaaaarrrrrrrrrrrrrrrrrrrrrrrrr" --pipeline=5

With list_experimental_zstd_dict_threshold=0 (disabled, default): used_memory_human:2.56GiB QPS: 1723935, P99 lat: 6ms
With list_experimental_zstd_dict_threshold=100000: used_memory_human: 32.84MiB, QPS: 1784274 P99 - 5.5ms

75 times less memory consumption for this synthetic use-case.
For real-world use-cases I expect to have 3-10x reduction in some cases.

Fixes #6899

Signed-off-by: Roman Gershman roman@dragonflydb.io

Copilot

Pull request overview

Implements an experimental ZSTD dictionary–based compression mode for QList to significantly reduce memory usage by sharing a per-thread dictionary state across all lists on the same shard thread.

Changes:

Adds a new list_experimental_zstd_dict_threshold flag and wires it into list promotion/creation to enable dict compression above a malloc-usage threshold.
Refactors QList compression flow into a dict-based 3-stage state machine (train → bulk-compress → incremental per-node compress) backed by thread-local dict state with explicit shutdown.
Adds ZSTD-related tests, including a new QListZstdTest suite.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/server/rdb_save.cc	Minor formatting-only change in list RDB save path.
src/server/list_family.cc	Adds and wires `list_experimental_zstd_dict_threshold` to configure `QList` dict compression threshold.
src/server/engine_shard.cc	Calls `QList::ShutdownThread()` during shard thread-local teardown to free TLS dict state.
src/core/zstd_test.cc	Adds a test exercising ZSTD CDict memory estimation API (and enables advanced API via macro).
src/core/qlist_test.cc	Adds a new `QListZstdTest` suite covering dict compression behaviors.
src/core/qlist.h	Introduces `ShutdownThread()` API, renames compression helper, and adds bitfields tracking dict compression state.
src/core/qlist.cc	Implements TLS-backed dict training/compression and integrates it into `CoolOff()` logic.

augmentcode · 2026-03-24T19:05:36Z

🤖 Augment PR Summary

Summary: Adds an experimental ZSTD dictionary-based compression mode for QList that shares a per-thread dictionary to significantly reduce memory usage.

Changes:

Introduce a thread-local ZstdDictState (CDict/DDict plus reusable CCtx/DCtx) shared by all lists on a shard thread.
Add per-list state bits to manage dict training, bulk compression, and steady-state per-node compression.
Train dictionaries using core/dict_builder helpers and compress interior nodes with ZSTD_compress_usingCDict.
Replace the prior recompress-only helper with RecompressNode() to support dict recompression after reads/mutations.
Wire a new flag --list_experimental_zstd_dict_threshold to enable/trigger dict training by list malloc usage.
Ensure thread-local dict cleanup via QList::ShutdownThread() on shard teardown and in tests.
Update RDB list serialization to avoid persisting ZSTD bytes as RDB-LZF by first decompressing ZSTD nodes to raw.
Add a QListZstdTest suite covering read, push/pop, index access, and incremental compression behavior.

Technical Notes: The dict mode is mutually exclusive with depth-based LZF compression and is synchronous (may block shard thread during training/bulk pass).

_{🤖 Was this summary useful? React with 👍 or 👎}

augmentcode

Review completed. 4 suggestions posted.

Comment augment review to trigger a new review at any time.

romange · 2026-03-25T07:39:52Z

augment review

Move ZstdDictState from per-QList unique_ptr to a thread-local raw pointer shared across all QList instances on the same thread. Motivation: ZSTD_CDict and ZSTD_DDict take 200-300KB each and holding them per list kills the main motivation to save memory. - Rename AllowCompression -> AllowLZFCompression - Split TrainAndCompressZstdDict into TrainZstdDict, CompressWithZstdDict, and CompressNodeWithDict - Add bit fields for tracking compression state: dict_learning_failed, dict_compress_failed, dict_bulk_finished - CoolOff ZSTD branch: 3-state machine (train -> bulk compress -> per-node compress) - Add ShutdownThread() to free thread-local dict on shard teardown - Wire list_experimental_zstd_dict_threshold flag in list_family - Add QListZstdTest suite with 6 tests Test: `./dfly_bench -n 80000 -p 6379 -qps=0 -d 64 --key_maximum=10 --command="lpush __key__ foooooooooobbbbbbbbbbbbbbaaaaaaaaaaaaaaaaaarrrrrrrrrrrrrrrrrrrrrrrrr" --pipeline=5` 1. With list_experimental_zstd_dict_threshold=0 (disabled, default): used_memory_human:2.56GiB QPS: 1723935, P99 lat: 6ms 2. With list_experimental_zstd_dict_threshold=100000: used_memory_human: 32.84MiB, QPS: 1784274 P99 - 5.5ms 75 times less memory consumption for this synthetic use-case. For real-world use-cases I expect to have 3-10x reduction in some cases. Signed-off-by: Roman Gershman <roman@dragonflydb.io>

augmentcode

Review completed. 2 suggestions posted.

Comment augment review to trigger a new review at any time.

augmentcode · 2026-03-25T07:45:58Z

+    ZSTD_DCtx_reset(tl_zstd_dict->dctx, ZSTD_reset_session_only);
+    size_t dsz = ZSTD_decompress_usingDDict(tl_zstd_dict->dctx, decompressed, node->sz,
+                                            lzf->compressed, lzf->sz, ddict);
+    if (ZSTD_isError(dsz) || dsz != node->sz) {


DecompressRaw() can now return false for ZSTD failures, but callers (via TryDecompressInternal()/AccessForReads()) don’t treat that as a hard stop and will likely keep using node->entry as RAW listpack bytes while it’s still compressed; also the compression stats are decremented before success is known. Consider making decompression failure fatal again or propagating the failure so callers can abort and stats/state can be kept consistent.

Severity: high

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}

@claude can you fix?

@claude[agent] fix by adding a new commit. do not change the PR description or its title.

Claude · 2026-03-25T15:32:29Z

augment review

Addressed review feedback in 73efcba:

Fixed brittle test assertion in EstimateDictSize (now checks for valid result and > 0)
Added recompression path for ZSTD-dict mode to properly handle nodes marked for recompression after reads

augmentcode

Review completed. 2 suggestions posted.

Comment augment review to trigger a new review at any time.

romange · 2026-03-26T06:28:08Z

augment review

augmentcode

Review completed. 1 suggestion posted.

Comment augment review to trigger a new review at any time.

vyavdoshenko

lgtm

Copilot AI review requested due to automatic review settings March 24, 2026 19:00

Copilot started reviewing on behalf of romange March 24, 2026 19:00 View session

Copilot AI reviewed Mar 24, 2026

View reviewed changes

Comment thread src/core/qlist.cc Outdated

Comment thread src/core/zstd_test.cc

Comment thread src/core/qlist_test.cc

Comment thread src/core/qlist.cc

Comment thread src/core/qlist.cc Outdated

augmentcode Bot reviewed Mar 24, 2026

View reviewed changes

Comment thread src/core/qlist.cc

Comment thread src/core/qlist.cc

Comment thread src/core/qlist.cc

Comment thread src/core/qlist.cc Outdated

romange force-pushed the List_Compr_POC branch from 2db0a65 to cb04233 Compare March 24, 2026 19:55

romange force-pushed the List_Compr_POC branch from cb04233 to f7fa796 Compare March 25, 2026 07:44

augmentcode Bot reviewed Mar 25, 2026

View reviewed changes

Claude started work on behalf of romange March 25, 2026 15:17 View session

Claude AI changed the title ~~chore(core): implement ZSTD dict based compression~~ Addressing PR comments Mar 25, 2026

Claude finished work on behalf of romange March 25, 2026 15:19

romange changed the title ~~Addressing PR comments~~ chore(core): implement ZSTD dict based compression Mar 25, 2026

Claude started work on behalf of romange March 25, 2026 15:24 View session

Claude AI changed the title ~~chore(core): implement ZSTD dict based compression~~ Addressing PR comments Mar 25, 2026

Claude finished work on behalf of romange March 25, 2026 15:33

augmentcode Bot reviewed Mar 25, 2026

View reviewed changes

Comment thread src/core/qlist.cc

Comment thread src/core/qlist.cc Outdated

romange changed the title ~~Addressing PR comments~~ chore(core): implement ZSTD dict based compression Mar 25, 2026

romange force-pushed the List_Compr_POC branch from 73efcba to f7fa796 Compare March 25, 2026 19:59

romange requested a review from vyavdoshenko March 26, 2026 06:28

augmentcode Bot reviewed Mar 26, 2026

View reviewed changes

Comment thread src/core/qlist.cc

vyavdoshenko approved these changes Mar 26, 2026

View reviewed changes

romange merged commit a90b685 into main Mar 26, 2026
26 checks passed

romange deleted the List_Compr_POC branch March 26, 2026 12:42

romange mentioned this pull request Mar 27, 2026

Add compression stats and ratio threshold for ZSTD dict #7000

Merged

romange mentioned this pull request May 5, 2026

feat(tools): add Copilot backend, per-commit cache, and async refactor to release notes generator #7264

Merged

Conversation

romange commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

augmentcode Bot commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

augmentcode Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

romange commented Mar 25, 2026

Uh oh!

augmentcode Bot left a comment

Choose a reason for hiding this comment

Uh oh!

augmentcode Bot Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

romange Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

romange Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Claude AI commented Mar 25, 2026

Uh oh!

augmentcode Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

romange commented Mar 26, 2026

Uh oh!

augmentcode Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vyavdoshenko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

romange commented Mar 24, 2026 •

edited

Loading

augmentcode Bot commented Mar 24, 2026 •

edited

Loading