diff --git a/CHANGELOG.md b/CHANGELOG.md index 81f5804..d92a8ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,13 @@ While the major version is 0, the public API may change in any release. ## [Unreleased] +### Documentation + +- The comparison benchmarks include theine and sturdyc, `make bench-compare` adds a serial + `Get` and five runs of each, and the README tables are medians of those runs. +- The README compares features with otter, theine, sturdyc and ttlcache, including what + they do that this library does not, and points to sturdyc for background refresh. + ## [0.7.0] - 2026-09-30 ### Added diff --git a/Makefile b/Makefile index d288c62..ddb53df 100644 --- a/Makefile +++ b/Makefile @@ -15,7 +15,7 @@ bench: .PHONY: bench-compare bench-compare: - cd benchmarks && go test -run '^$$' -bench 'Get$$|Set$$|Mixed$$' -benchmem ./... + cd benchmarks && go test -run '^$$' -bench 'Get$$|GetSerial$$|Set$$|Mixed$$' -benchmem -count=5 ./... cd benchmarks && go test -run '^$$' -bench HitRate -benchtime=300000x ./... .PHONY: cover diff --git a/README.md b/README.md index 1db38f3..319586e 100644 --- a/README.md +++ b/README.md @@ -493,36 +493,39 @@ calling it is still better than relying on when the collector gets around to it. The `benchmarks/` module measures this cache against the ones it would otherwise be replacing. It is a separate module so the root stays dependency-free; `make bench-compare` -runs it. Same machine as above, 256-byte values, a budget sized to 4096 of them. The -sanecache rows are its knobs, each one added to the row above it. +runs it. Same machine as above, Go 1.24, 256-byte values, a budget sized to 4096 of them, +medians of five runs. The sanecache rows are its knobs, each one added to the row above it. | ns/op | serial `Get` | `Get` ×8 | `Set` ×8 | 90/10 ×8 | |---|---:|---:|---:|---:| -| sanecache, defaults | 61 | 139 | 226 | 151 | -| ⤷ `Shards: 16` | 64 | 42 | 91 | 49 | -| ⤷ + `ClockGranularity` | 34 | 32 | 84 | 42 | -| ⤷ + `ClearOnFull` | 32 | 20 | 90 | 39 | -| [otter](https://github.com/maypok86/otter) v2 | 77 | 14 | 263 | 30 | -| [ristretto](https://github.com/dgraph-io/ristretto) v2 | 86 | 21 | 324 | 65 | -| [golang-lru](https://github.com/hashicorp/golang-lru) `v2/expirable` | 52 | 142 | 190 | 165 | -| [ttlcache](https://github.com/jellydator/ttlcache) v3 | 63 | 177 | 267 | 192 | +| sanecache, defaults | 62 | 141 | 229 | 156 | +| ⤷ `Shards: 16` | 64 | 42 | 90 | 49 | +| ⤷ + `ClockGranularity` | 33 | 32 | 81 | 38 | +| ⤷ + `ClearOnFull` | 31 | 19 | 86 | 38 | +| [otter](https://github.com/maypok86/otter) v2 | 80 | 14 | 268 | 29 | +| [theine](https://github.com/Yiling-J/theine-go) | 91 | 13 | 174 | 57 | +| [ristretto](https://github.com/dgraph-io/ristretto) v2 | 87 | 21 | 252 | 70 | +| [sturdyc](https://github.com/viccon/sturdyc) | 62 | 32 | 121 | 67 | +| [golang-lru](https://github.com/hashicorp/golang-lru) `v2/expirable` | 54 | 140 | 185 | 168 | +| [ttlcache](https://github.com/jellydator/ttlcache) v3 | 63 | 175 | 264 | 193 | The first column is what one lookup costs; the second is what eight goroutines get out of the same machine. Read them together, because they say opposite things. Per operation this -cache is *cheaper* than otter and ristretto — they spend their time maintaining a frequency -sketch and a set of ring buffers that only pay off later. What they buy with it is scaling: -otter turns eight cores into 5.4x the throughput, this one into 1.6x, and unsharded into -less than 1x, because reads take a lock and locks are where cores queue. +cache is *cheaper* than otter, theine and ristretto — they spend their time maintaining a +frequency sketch and a set of ring buffers that only pay off later. What they buy with it is +scaling: otter and theine turn eight cores into 6x to 7x the throughput, this one into 1.6x, +and unsharded into less than 1x, because reads take a lock and locks are where cores queue. So the honest shape of it is not "slower". It is: **the same work per operation, and less of the machine used to do it in parallel.** Whether that matters is a question about the -read rate. At one lookup per request, the gap between 20 ns and 14 ns is six nanoseconds +read rate. At one lookup per request, the gap between 19 ns and 13 ns is six nanoseconds against a request budget measured in milliseconds. It starts to matter when one request does thousands of lookups, or when the cache more or less is the service. -Writes are the other way round: 91 ns against 190 to 324. An admission policy has to decide -whether to accept each write and update its sketch, and that costs more than taking a lock -does. A write-heavy cache is the case where this library is simply faster. +Writes are the other way round: 81 to 90 ns against 121 to 268. An admission policy has to +decide whether to accept each write and update its sketch, and that costs more than taking a +lock does; sturdyc, closest here, has no admission policy either. A write-heavy cache is the +case where this library is simply faster. The number that usually matters more than any of those is how much of a fixed budget each policy turns into hits. 4096 entries against a key space of 100,000: @@ -530,19 +533,50 @@ policy turns into hits. 4096 entries against a key space of 100,000: | %hit | zipf s=1.20 | zipf s=1.01 | zipf + scan | |---|---:|---:|---:| | sanecache (LRU) | 87.3 | 65.8 | 78.0 | -| sanecache (ClearOnFull) | 83.1 | 58.0 | 73.7 | -| otter v2 | 88.9 | **71.4** | 80.8 | -| ristretto v2 | 87.4 | 68.7 | 78.9 | +| sanecache (ClearOnFull) | 83.2 | 57.9 | 73.7 | +| otter v2 | 88.9 | **71.3** | **80.8** | +| theine | **89.0** | **71.3** | 80.7 | +| ristretto v2 | 87.3 | 68.7 | 79.0 | +| sturdyc | 84.9 | 61.6 | 75.3 | | golang-lru `expirable` | 87.3 | 65.8 | 78.0 | | ttlcache v3 | 87.3 | 65.8 | 78.0 | -When the hot set fits, every policy looks the same and the extra machinery buys 1.6 points. -When it does not, W-TinyLFU is worth 5.6 points of hit rate over LRU — and 5.6 points of +When the hot set fits, every policy looks the same and the extra machinery buys 1.7 points. +When it does not, W-TinyLFU is worth 5.5 points of hit rate over LRU — and 5.5 points of upstream traffic is worth more than every nanosecond in the table above it. That is the real -reason to pick otter, and it is a better one than throughput. +reason to pick otter or theine, and it is a better one than throughput. -The same table prices the cheap read lock: `ClearOnFull` buys its 40% by giving up 7.8 +The same table prices the cheap read lock: `ClearOnFull` buys its 40% by giving up 7.9 points of hit rate. It is a trade for caches whose access order is flat, not a free win. +sturdyc makes a milder version of the same trade: when a shard is full it drops the +least recently used tenth of it at once, which costs it 4 points against a plain LRU. + +### The features are not the difference either + +Loading a cold key once, loading many keys in one call and a budget in something other +than entries are not unique to this library. As of otter v2.3.0, +[theine](https://github.com/Yiling-J/theine-go) v0.6.2, +[sturdyc](https://github.com/viccon/sturdyc) v1.1.6 and ttlcache v3.4.1: + +| | one load per key | batch load | budget by cost | "not found" remembered | value over budget | +|---|---|---|---|---|---| +| sanecache | `GetOrLoad` | `GetManyOrLoad` | `MaxBytes` + `Cost` | `SetNegative` | `ErrTooLarge` | +| otter v2 | `Get` + `Loader` | `BulkGet` | `MaximumWeight` + `Weigher` | no: `ErrNotFound` deletes | accepted, then evicted | +| theine | loading cache | — | `Cost` | — | `Set` returns `false` | +| sturdyc | `GetOrFetch` | `GetOrFetchBatch` | — | `WithMissingRecordStorage` | — | +| ttlcache v3 | `SuppressedLoader` | — | `WithMaxCost` | — | evicts everything, itself last | + +They also do things this library does not. otter and sturdyc refresh entries in the +background before they expire, so a hot key never goes cold; otter and theine can save the +cache to a file and load it back; sturdyc coalesces refreshes into batches and can sit in +front of a distributed store. sturdyc can also hold several value types in one cache, with +typed package-level functions on top, though its capacity is a count of entries rather than +bytes. + +What is left is the combination rather than any single feature: a byte budget that refuses +what does not fit, missing records charged against that budget, and value types with costs +of their own [under one budget](#several-value-types-under-one-budget). If background +refresh matters more to you than that, otter or sturdyc is the better fit. ## When to use something else @@ -550,8 +584,11 @@ points of hit rate. It is a trade for caches whose access order is flat, not a f access-frequency estimation are worth the complexity → [otter](https://github.com/maypok86/otter). The hit-rate table above is the size of the prize, and it is the biggest number on this page. - Your read path is hot enough that how well a cache scales across cores is a real - difference → otter or [ristretto](https://github.com/dgraph-io/ristretto), and budget for - `Wait()` in the tests. + difference → otter or [ristretto](https://github.com/dgraph-io/ristretto); with ristretto, + budget for `Wait()` in the tests. +- You call an upstream API and want entries refreshed before they expire, refreshes + coalesced into batches, or a distributed store behind the cache → + [sturdyc](https://github.com/viccon/sturdyc). - You are caching hundreds of megabytes and GC pressure from millions of live pointers is your actual problem → [bigcache](https://github.com/allegro/bigcache) or [freecache](https://github.com/coocood/freecache), which reduce GC scanning by storing entries @@ -564,7 +601,7 @@ points of hit rate. It is a trade for caches whose access order is flat, not a f ## Status -v0.6. The API above is what exists and is tested; expect it to move before v1. +v0.7. The API above is what exists and is tested; expect it to move before v1. Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for what this library optimises for before proposing a change. diff --git a/benchmarks/compare_test.go b/benchmarks/compare_test.go index 0ad0200..304c1e2 100644 --- a/benchmarks/compare_test.go +++ b/benchmarks/compare_test.go @@ -19,11 +19,13 @@ import ( "testing" "time" + "github.com/Yiling-J/theine-go" "github.com/andared/sanecache" "github.com/dgraph-io/ristretto/v2" "github.com/hashicorp/golang-lru/v2/expirable" "github.com/jellydator/ttlcache/v3" "github.com/maypok86/otter/v2" + "github.com/viccon/sturdyc" ) const ( @@ -61,9 +63,11 @@ func factories() []factory { return openSane(16, time.Millisecond, sanecache.ClearOnFull) }}, {"otter", openOtter}, + {"theine", openTheine}, {"ristretto", openRistretto}, {"golang-lru-expirable", openExpirable}, {"ttlcache", openTTLCache}, + {"sturdyc", openSturdyc}, } } @@ -108,6 +112,28 @@ func (o otterCache) wait() {} func (o otterCache) close() {} +type theineCache struct{ c *theine.Cache[string, []byte] } + +func openTheine() cache { + c, err := theine.NewBuilder[string, []byte](capacity * valueSize). + Cost(func(v []byte) int64 { return int64(len(v)) }). + Build() + if err != nil { + panic(err) + } + + return theineCache{c} +} + +func (t theineCache) get(key string) bool { _, ok := t.c.Get(key); return ok } + +// Cost 0 asks theine to call the Cost function given to the builder. +func (t theineCache) set(key string, v []byte) { t.c.SetWithTTL(key, v, 0, benchTTL) } + +func (t theineCache) wait() {} + +func (t theineCache) close() { t.c.Close() } + type ristrettoCache struct { c *ristretto.Cache[string, []byte] } @@ -180,6 +206,26 @@ func (t ttlCache) wait() {} func (t ttlCache) close() { t.c.Stop() } +type sturdycCache struct{ c *sturdyc.Client[[]byte] } + +func openSturdyc() cache { + // Entry count rather than bytes: this cache has no notion of cost. Shards + // and the share evicted when full are the values from its own README. + // Its expiry sweeper never exits and keeps the client alive, so every run + // would leak a full cache into the ones after it; nothing expires mid-run, + // so turning it off changes nothing else. + return sturdycCache{sturdyc.New[[]byte](capacity, 10, benchTTL, 10, + sturdyc.WithNoContinuousEvictions())} +} + +func (s sturdycCache) get(key string) bool { _, ok := s.c.Get(key); return ok } + +func (s sturdycCache) set(key string, v []byte) { s.c.Set(key, v) } + +func (s sturdycCache) wait() {} + +func (s sturdycCache) close() {} + // keySet precomputes keys so that the benchmarks measure the cache rather than // strconv. func keySet(n int) []string { diff --git a/benchmarks/go.mod b/benchmarks/go.mod index 77d8755..fa361e0 100644 --- a/benchmarks/go.mod +++ b/benchmarks/go.mod @@ -7,19 +7,23 @@ go 1.24.0 replace github.com/andared/sanecache => ../ require ( + github.com/Yiling-J/theine-go v0.6.2 github.com/andared/sanecache v0.1.0 github.com/dgraph-io/ristretto/v2 v2.4.2 github.com/hashicorp/golang-lru/v2 v2.0.7 github.com/jellydator/ttlcache/v3 v3.4.1 github.com/maypok86/otter/v2 v2.3.0 + github.com/viccon/sturdyc v1.1.6 ) require ( github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/davecgh/go-spew v1.1.1 // indirect github.com/dustin/go-humanize v1.0.1 // indirect + github.com/klauspost/cpuid/v2 v2.0.9 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect github.com/stretchr/testify v1.11.1 // indirect + github.com/zeebo/xxh3 v1.0.2 // indirect golang.org/x/sync v0.16.0 // indirect golang.org/x/sys v0.36.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect diff --git a/benchmarks/go.sum b/benchmarks/go.sum index 08e8489..d38a07e 100644 --- a/benchmarks/go.sum +++ b/benchmarks/go.sum @@ -1,3 +1,5 @@ +github.com/Yiling-J/theine-go v0.6.2 h1:1GeoXeQ0O0AUkiwj2S9Jc0Mzx+hpqzmqsJ4kIC4M9AY= +github.com/Yiling-J/theine-go v0.6.2/go.mod h1:08QpMa5JZ2pKN+UJCRrCasWYO1IKCdl54Xa836rpmDU= github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= @@ -8,23 +10,38 @@ github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da h1:aIftn67I1fkbMa5 github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= +github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= +github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= github.com/jellydator/ttlcache/v3 v3.4.1 h1:bOdXmXiycyK6E6Qjyuj5vl+/vU3SCOoDs8a86NbHjAQ= github.com/jellydator/ttlcache/v3 v3.4.1/go.mod h1:j7LO12PNghFg5+0v9budMAT4rDK4JY969jb9vOdOBBk= +github.com/klauspost/cpuid/v2 v2.0.9 h1:lgaqFMSdTdQYdZ04uHyN2d/eKdOMyi2YLSvlQIBFYa4= +github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= +github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= +github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= github.com/maypok86/otter/v2 v2.3.0 h1:8H8AVVFUSzJwIegKwv1uF5aGitTY+AIrtktg7OcLs8w= github.com/maypok86/otter/v2 v2.3.0/go.mod h1:XgIdlpmL6jYz882/CAx1E4C1ukfgDKSaw4mWq59+7l8= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +github.com/viccon/sturdyc v1.1.6 h1:ISrV4YrmkakI1iON09gYNF4YA5gkR3ZOC5EErbJ3S8Q= +github.com/viccon/sturdyc v1.1.6/go.mod h1:OCBEgG/i48uugKQ498UQlfMHmf5j8MYY8a4BApfVnMo= +github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ= +github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0= +github.com/zeebo/xxh3 v1.0.2 h1:xZmwmqxHZA8AI603jOQ0tMqmBr9lPeFwGg6d+xy9DC0= +github.com/zeebo/xxh3 v1.0.2/go.mod h1:5NWz9Sef7zIDm2JHfFlcQvNekmcEl9ekUZQQKCYaDcA= go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw= golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA= golang.org/x/sys v0.36.0 h1:KVRy2GtZBrk1cBYA7MKu5bEZFxQk4NIDV6RLVcC8o0k= golang.org/x/sys v0.36.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks= -gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=