From e213e8e87c419d0cec609e6cb9f32631eda17b2e Mon Sep 17 00:00:00 2001 From: Andrey Ryabokon Date: Fri, 9 Oct 2026 05:44:48 +0300 Subject: [PATCH 1/9] Refresh values in the background before they expire --- CHANGELOG.md | 17 ++ README.md | 84 +++++-- bench_test.go | 25 ++ cache.go | 191 +++++++++++---- loader.go | 61 +++-- loader_batch.go | 69 +++++- refresh_test.go | 611 ++++++++++++++++++++++++++++++++++++++++++++++++ shard.go | 65 +++++- stats.go | 52 +++-- view.go | 117 +++++++--- view_loader.go | 84 ++++--- 11 files changed, 1204 insertions(+), 172 deletions(-) create mode 100644 refresh_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index d92a8ac..5898f49 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,23 @@ While the major version is 0, the public API may change in any release. ## [Unreleased] +### Added + +- `Options.RefreshAfter` and `ViewOptions.RefreshAfter`: a value read after it is this old + is returned at once and reloaded in the background, so a key in steady use is replaced + before it expires instead of making the next caller wait for the upstream. `GetOrLoad`, + `GetManyOrLoad` and `GetOrLoadFunc` refresh with their loaders, `Get` and `Lookup` with + the loaders in the options. Each value gets one refresh; a failed one leaves it in place + until its TTL, and the TTL is never extended. `GetManyOrLoad` refreshes the keys it finds + due in one `BatchLoader` call. Off by default; it must be shorter than the TTL. +- `Stats.Refreshes`, `Stats.RefreshErrors` and the same in `ViewStats`. Refreshes are not + counted in `Loads`. + +### Changed + +- `Stats` and `ViewStats` gained fields. Code that builds them with positional fields needs + updating; code that reads fields by name does not. + ### Documentation - The comparison benchmarks include theine and sturdyc, `make bench-compare` adds a serial diff --git a/README.md b/README.md index 319586e..abc01c1 100644 --- a/README.md +++ b/README.md @@ -276,6 +276,50 @@ concurrently, one call each. `Stats().Batches` counts `BatchLoader` calls; again `Loads`, which counts keys, it says how many keys an upstream call carries. Batch loading is not available on views yet. +## Refreshing before expiry + +A TTL is a promise about staleness, and a cache keeps it by going back to the upstream. On +a key in steady use it does that on the request path: the value expires, the next caller +misses and waits, and every key the service relies on goes cold once per TTL. In a busy +cache with a high hit rate, almost every miss can be exactly that — `Expirations` and +`Loads` moving in step is the sign of it. + +`RefreshAfter` moves that load off the request path: + +```go +c := sanecache.New(sanecache.Options[int, *Rule]{ + TTL: time.Minute, + RefreshAfter: 40 * time.Second, + Loader: loadRule, +}) +``` + +A read that finds a value older than `RefreshAfter` returns it at once and starts a load +in the background. The value in the cache is replaced when the load succeeds, and the TTL +starts over. A key nobody reads is not refreshed: it expires as it did before, so refresh +spends upstream calls only on keys that are still wanted. + +**One refresh per value.** However many reads see a value due, it gets one load. The +refresh takes part in single flight like any load: a caller that finds the key expired +before the refresh finishes waits for it instead of starting another, and a `Delete` or a +successful write keeps its result out. + +**The TTL still holds.** A failed refresh leaves the value in place until its TTL runs out, +and is not retried; after that the next caller loads the key and sees the error, as it +would without refresh. An upstream that is down therefore shows up one TTL later than it +otherwise would, and never later than that. A refresh answered with `ErrNotFound` turns the +value into a negative entry when `NegativeTTL` is set. + +**Any read with a loader refreshes.** `GetOrLoad` and `GetManyOrLoad` use the loaders in +`Options`, `GetOrLoadFunc` the function it was given, and `Get` and `Lookup` the loaders in +`Options` when there are any. `GetManyOrLoad` sends the keys it finds due to the +`BatchLoader` in one call of their own, apart from the keys it is waiting for. Views take a +`RefreshAfter` of their own and refresh with their own loaders. + +Jitter moves the refresh along with the expiry it belongs to. `Stats().Refreshes` counts +finished refreshes and `RefreshErrors` the failed ones; neither is part of `Loads`, which +stays what misses cost. + ## Several value types under one budget A budget in bytes is only worth having if it covers everything competing for the memory. @@ -460,7 +504,7 @@ metrics interval and export it however you like: s := c.Stats() // s.Hits, s.Misses, s.Negatives, s.TypeMisses, s.Evictions, s.Expirations, // s.Replacements, s.Rejections, s.Loads, s.LoadNotFound, s.LoadErrors, s.Coalesced, s.Batches, -// s.Entries, s.Bytes, s.HitRate() +// s.Refreshes, s.RefreshErrors, s.Entries, s.Bytes, s.HitRate() ``` `ViewStats()` returns the same kind of snapshot for every view opened on the cache, keyed @@ -469,7 +513,7 @@ by view name, so an exporter needs no list of views of its own: ```go for name, vs := range c.ViewStats() { // vs.Hits, vs.Misses, vs.Negatives, vs.TypeMisses, vs.Loads, vs.LoadNotFound, - // vs.LoadErrors, vs.Coalesced, vs.HitRate() + // vs.LoadErrors, vs.Coalesced, vs.Refreshes, vs.RefreshErrors, vs.HitRate() } ``` @@ -553,30 +597,28 @@ least recently used tenth of it at once, which costs it 4 points against a plain ### The features are not the difference either -Loading a cold key once, loading many keys in one call and a budget in something other -than entries are not unique to this library. As of otter v2.3.0, +Loading a cold key once, loading many keys in one call, refreshing ahead of expiry and a +budget in something other than entries are not unique to this library. As of otter v2.3.0, [theine](https://github.com/Yiling-J/theine-go) v0.6.2, [sturdyc](https://github.com/viccon/sturdyc) v1.1.6 and ttlcache v3.4.1: -| | one load per key | batch load | budget by cost | "not found" remembered | value over budget | -|---|---|---|---|---|---| -| sanecache | `GetOrLoad` | `GetManyOrLoad` | `MaxBytes` + `Cost` | `SetNegative` | `ErrTooLarge` | -| otter v2 | `Get` + `Loader` | `BulkGet` | `MaximumWeight` + `Weigher` | no: `ErrNotFound` deletes | accepted, then evicted | -| theine | loading cache | — | `Cost` | — | `Set` returns `false` | -| sturdyc | `GetOrFetch` | `GetOrFetchBatch` | — | `WithMissingRecordStorage` | — | -| ttlcache v3 | `SuppressedLoader` | — | `WithMaxCost` | — | evicts everything, itself last | - -They also do things this library does not. otter and sturdyc refresh entries in the -background before they expire, so a hot key never goes cold; otter and theine can save the -cache to a file and load it back; sturdyc coalesces refreshes into batches and can sit in -front of a distributed store. sturdyc can also hold several value types in one cache, with -typed package-level functions on top, though its capacity is a count of entries rather than -bytes. +| | one load per key | batch load | refresh ahead | budget by cost | "not found" remembered | value over budget | +|---|---|---|---|---|---|---| +| sanecache | `GetOrLoad` | `GetManyOrLoad` | `RefreshAfter` | `MaxBytes` + `Cost` | `SetNegative` | `ErrTooLarge` | +| otter v2 | `Get` + `Loader` | `BulkGet` | `RefreshCalculator` | `MaximumWeight` + `Weigher` | no: `ErrNotFound` deletes | accepted, then evicted | +| theine | loading cache | — | — | `Cost` | — | `Set` returns `false` | +| sturdyc | `GetOrFetch` | `GetOrFetchBatch` | `WithEarlyRefreshes` | — | `WithMissingRecordStorage` | — | +| ttlcache v3 | `SuppressedLoader` | — | — | `WithMaxCost` | — | evicts everything, itself last | + +They also do things this library does not. otter and theine can save the cache to a file +and load it back; sturdyc retries failed refreshes, coalesces refreshes from many requests +into batches and can sit in front of a distributed store. sturdyc can also hold several +value types in one cache, with typed package-level functions on top, though its capacity is +a count of entries rather than bytes. What is left is the combination rather than any single feature: a byte budget that refuses what does not fit, missing records charged against that budget, and value types with costs -of their own [under one budget](#several-value-types-under-one-budget). If background -refresh matters more to you than that, otter or sturdyc is the better fit. +of their own [under one budget](#several-value-types-under-one-budget). ## When to use something else @@ -586,7 +628,7 @@ refresh matters more to you than that, otter or sturdyc is the better fit. - Your read path is hot enough that how well a cache scales across cores is a real difference → otter or [ristretto](https://github.com/dgraph-io/ristretto); with ristretto, budget for `Wait()` in the tests. -- You call an upstream API and want entries refreshed before they expire, refreshes +- You call an upstream API and want refreshes retried, refreshes from many requests coalesced into batches, or a distributed store behind the cache → [sturdyc](https://github.com/viccon/sturdyc). - You are caching hundreds of megabytes and GC pressure from millions of live pointers is diff --git a/bench_test.go b/bench_test.go index ca7b9e4..fc43049 100644 --- a/bench_test.go +++ b/bench_test.go @@ -145,6 +145,31 @@ func BenchmarkGetOrLoad(b *testing.B) { } } +// BenchmarkGetOrLoadRefreshAfter is BenchmarkGetOrLoad with RefreshAfter set +// and nothing due yet, which is where a refreshing cache spends nearly all its +// reads: what it adds is the check of whether the value is due. +func BenchmarkGetOrLoadRefreshAfter(b *testing.B) { + c := New(Options[int, int]{ + TTL: time.Hour, + RefreshAfter: 30 * time.Minute, + MaxBytes: benchKeys, + Cost: unitCost[int](), + Loader: func(_ context.Context, key int) (int, error) { + return key, nil + }, + }) + defer c.Close() + for i := range benchKeys { + _ = c.Set(i, i) + } + + ctx := context.Background() + b.ResetTimer() + for i := range b.N { + _, _ = c.GetOrLoad(ctx, i%benchKeys) + } +} + // BenchmarkGetManyOrLoad is the warm path of a batch: every key is cached, so // the loader never runs and what is measured is the lookup of the batch and the // map it comes back in, against the same keys fetched one GetOrLoad at a time. diff --git a/cache.go b/cache.go index b2c7bd9..a680bc5 100644 --- a/cache.go +++ b/cache.go @@ -107,6 +107,30 @@ type Options[K comparable, V any] struct { // so keys written together do not expire together. Must be 0..100. Jitter int + // RefreshAfter starts reloading a value in the background once it is this + // old, so that a key in steady use is replaced before it expires instead of + // going cold and making the next caller wait for the upstream. Zero, the + // default, turns it off. It must be shorter than TTL: a value is still never + // served past its TTL, so the upstream being down shows up as it does + // without refresh, one TTL later. + // + // The read that finds a value due returns it at once and starts the load: + // GetOrLoad, GetManyOrLoad and GetOrLoadFunc with their loaders, Get and + // Lookup with Loader or BatchLoader when either is set. A due value gets one + // refresh, whichever read sees it first. If that load fails, the value stays + // until its TTL runs out and the caller after that loads it the usual way; + // a refresh that reports ErrNotFound replaces the value with a negative + // entry when NegativeTTL is set, and leaves it to its TTL otherwise. + // GetManyOrLoad refreshes the keys it finds due in one BatchLoader call. + // + // A refresh is a load like any other: it shares single flight with the key's + // other loads, so a caller that finds the key expired before the refresh + // finishes waits for it rather than starting another, and a Delete, a Clear + // or a successful write keeps its result out. Its context carries the values + // of the read that started it and is never cancelled. Jitter moves the + // refresh with the expiry it belongs to. + RefreshAfter time.Duration + // MaxBytes is the total budget in bytes, split evenly across shards. It // requires Cost. Zero means unbounded. MaxBytes int64 @@ -215,17 +239,22 @@ type core[K comparable, V any] struct { mask uint64 seed maphash.Seed - ttl time.Duration - negativeTTL time.Duration - jitter int64 - cost func(V) int64 - onEvict func(K, V, EvictReason) - countStats bool + ttl time.Duration + negativeTTL time.Duration + refreshAfter time.Duration + jitter int64 + cost func(V) int64 + onEvict func(K, V, EvictReason) + countStats bool loader func(context.Context, K) (V, error) batchLoader func(context.Context, []K) (map[K]V, error) flights []*flightGroup[K, V] + // getRefresher is refresherWith(loader), worked out once: it is asked on + // every Get. + getRefresher refresher + // views holds one set of counters per view name. Views with the same name // share a namespace in storage, so they share counters too; the map only // grows with the number of distinct names, which is fixed by the program. @@ -257,25 +286,31 @@ func New[K comparable, V any](o Options[K, V]) *Cache[K, V] { panic("sanecache: TTL and NegativeTTL must not be negative") case o.ClockGranularity < 0: panic("sanecache: ClockGranularity must not be negative") + case o.RefreshAfter < 0: + panic("sanecache: RefreshAfter must not be negative") + case o.RefreshAfter > 0 && (o.TTL == 0 || o.RefreshAfter >= o.TTL): + panic(fmt.Sprintf("sanecache: RefreshAfter must be shorter than TTL, got %v with TTL %v", o.RefreshAfter, o.TTL)) } n := shardCount(o.Shards) cr := &core[K, V]{ - shards: make([]*shard[K, V], n), - mask: uint64(n - 1), - seed: maphash.MakeSeed(), - ttl: o.TTL, - negativeTTL: o.NegativeTTL, - jitter: int64(o.Jitter), - cost: o.Cost, - onEvict: o.OnEvict, - countStats: !o.DisableStats, - loader: o.Loader, - batchLoader: o.BatchLoader, + shards: make([]*shard[K, V], n), + mask: uint64(n - 1), + seed: maphash.MakeSeed(), + ttl: o.TTL, + negativeTTL: o.NegativeTTL, + refreshAfter: o.RefreshAfter, + jitter: int64(o.Jitter), + cost: o.Cost, + onEvict: o.OnEvict, + countStats: !o.DisableStats, + loader: o.Loader, + batchLoader: o.BatchLoader, } if cr.loader == nil && cr.batchLoader != nil { cr.loader = cr.loadAsBatch } + cr.getRefresher = cr.refresherWith(cr.loader) perBytes := divideBudget(o.MaxBytes, int64(n)) perEntries := int(divideBudget(int64(o.MaxEntries), int64(n))) @@ -323,9 +358,12 @@ func (c *Cache[K, V]) Get(key K) (V, bool) { return v, st == StatusHit } -// Lookup returns the cached value and how the cache answered. +// Lookup returns the cached value and how the cache answered. With +// RefreshAfter and a loader in Options, a value due for a refresh starts one. func (c *Cache[K, V]) Lookup(key K) (V, Status) { - v, st, _ := c.core.lookup(key) + // One call, so that Get and Lookup still inline into their callers: + // starting the refresh here would make this too big for that. + v, st := c.core.read(key) return v, st } @@ -340,7 +378,7 @@ func (c *Cache[K, V]) Set(key K, value V) error { // means the entry never expires on its own. Successful writes supersede outstanding // loads for the key. func (c *Cache[K, V]) SetTTL(key K, value V, ttl time.Duration) error { - return c.core.setValue(key, value, c.core.valueCost(value), ttl) + return c.core.setValue(key, value, c.core.valueCost(value), ttl, c.core.refreshAfter, refreshCache) } // SetNegative records that the upstream reports no such key, for the configured @@ -446,13 +484,21 @@ func (c *core[K, V]) valueCost(v V) int64 { return c.cost(v) } -func (c *core[K, V]) setValue(key K, value V, cost int64, ttl time.Duration, tokens ...*loadToken) error { - return c.store(&entry[K, V]{ - key: key, - value: value, - cost: cost, - expiresAt: c.expiryAt(ttl), - }, tokens...) +// setValue stores a value that expires after ttl. Once it is refreshAfter old, a +// read by owner may refresh it; zero, or anything not shorter than ttl, means +// never. +func (c *core[K, V]) setValue( + key K, value V, cost int64, ttl, refreshAfter time.Duration, owner refresher, tokens ...*loadToken, +) error { + e := &entry[K, V]{key: key, value: value, cost: cost, byView: owner == refreshView} + var refreshAt int64 + e.expiresAt, refreshAt = c.deadlines(ttl, refreshAfter) + // A fresh entry already holds zero, and an atomic store is not free. + if refreshAt != 0 { + e.refreshAt.Store(refreshAt) + } + + return c.store(e, tokens...) } func (c *core[K, V]) setNegative(key K, ttl time.Duration, tokens ...*loadToken) error { @@ -503,27 +549,56 @@ func (c *core[K, V]) store(e *entry[K, V], tokens ...*loadToken) error { return nil } -// lookup is Cache.Lookup with the shard it landed on, which a view needs in -// order to count a type miss where the rest of that key's counters live. -func (c *core[K, V]) lookup(key K) (V, Status, *shard[K, V]) { +// lookup is a read with the shard it landed on, which a view needs in order to +// count a type miss where the rest of that key's counters live, and with whether +// the read, being by, claimed a refresh it now has to start. +func (c *core[K, V]) lookup(key K, by refresher) (V, Status, *shard[K, V], bool) { s := c.shardFor(key) - v, st, wasExpired := s.get(key, c.now()) - - if c.countStats { - switch st { - case StatusHit: - s.counters.hits.Add(1) - case StatusNegative: - s.counters.negatives.Add(1) - default: - s.counters.misses.Add(1) - if wasExpired { - s.counters.expirations.Add(1) - } + v, st, wasExpired, refresh := s.get(key, c.now(), by) + c.countLookup(s, st, wasExpired) + + return v, st, s, refresh +} + +// read is lookup for Get and Lookup, starting the refresh it claims with the +// loader in Options. It repeats lookup rather than calling it: neither inlines, +// and the extra call cost 4 ns of a 20 ns hit under a coarse clock. +func (c *core[K, V]) read(key K) (V, Status) { + s := c.shardFor(key) + v, st, wasExpired, refresh := s.get(key, c.now(), c.getRefresher) + c.countLookup(s, st, wasExpired) + if refresh { + c.refresh(background, key, c.loader) + } + + return v, st +} + +func (c *core[K, V]) countLookup(s *shard[K, V], st Status, wasExpired bool) { + if !c.countStats { + return + } + switch st { + case StatusHit: + s.counters.hits.Add(1) + case StatusNegative: + s.counters.negatives.Add(1) + default: + s.counters.misses.Add(1) + if wasExpired { + s.counters.expirations.Add(1) } } +} - return v, st, s +// refresherWith is who a read is when it would refresh with loader: the cache, +// or nobody when refresh is off or there is nothing to load with. +func (c *core[K, V]) refresherWith(loader func(context.Context, K) (V, error)) refresher { + if c.refreshAfter > 0 && loader != nil { + return refreshCache + } + + return refreshNone } func (c *core[K, V]) shardIndex(key K) uint64 { @@ -552,21 +627,39 @@ func (c *core[K, V]) now() int64 { // expiryAt turns a TTL into an absolute deadline, spreading it by Jitter percent. func (c *core[K, V]) expiryAt(ttl time.Duration) int64 { + expiresAt, _ := c.deadlines(ttl, 0) + + return expiresAt +} + +// deadlines is expiryAt along with the time to refresh. The refresh sits the same +// fraction of the way through the entry's life whatever Jitter made of it, so it +// moves with the expiry and never lands after it. +func (c *core[K, V]) deadlines(ttl, refreshAfter time.Duration) (expiresAt, refreshAt int64) { if ttl <= 0 { - return 0 + return 0, 0 } + life := ttl if c.jitter > 0 { if span := int64(ttl) * c.jitter / 100; span > 0 { - ttl += time.Duration(rand.Int64N(2*span+1) - span) + life += time.Duration(rand.Int64N(2*span+1) - span) } // Full jitter can land on zero, which would mean "never expires". - if ttl <= 0 { - ttl = 1 + if life <= 0 { + life = 1 } } - return c.now() + int64(ttl) + now := c.now() + expiresAt = now + int64(life) + if refreshAfter > 0 && refreshAfter < ttl { + // In floating point because the exact product of two durations overflows. + at := int64(float64(refreshAfter) * (float64(life) / float64(ttl))) + refreshAt = now + max(at, 1) + } + + return expiresAt, refreshAt } func (c *core[K, V]) notify(e *entry[K, V], r EvictReason) { diff --git a/loader.go b/loader.go index 0799df5..bb19982 100644 --- a/loader.go +++ b/loader.go @@ -29,7 +29,7 @@ func (c *Cache[K, V]) GetOrLoad(ctx context.Context, key K) (V, error) { return zero, ErrNoLoader } - if v, done, err := c.cached(ctx, key); done { + if v, done, err := c.cached(ctx, key, c.core.loader); done { return v, err } @@ -57,17 +57,24 @@ func (c *Cache[K, V]) GetOrLoadFunc(ctx context.Context, key K, load func(contex return zero, ErrNoLoader } - if v, done, err := c.cached(ctx, key); done { + loader := func(ctx context.Context, _ K) (V, error) { return load(ctx) } + if v, done, err := c.cached(ctx, key, loader); done { return v, err } - return c.core.load(ctx, key, func(ctx context.Context, _ K) (V, error) { return load(ctx) }) + return c.core.load(ctx, key, loader) } -// cached answers a load from the cache when it can. done is false only when the -// key is a miss and the caller still has time to wait for a load. -func (c *Cache[K, V]) cached(ctx context.Context, key K) (v V, done bool, err error) { - switch v, st := c.Lookup(key); st { +// cached answers a load from the cache when it can, starting a refresh with +// loader when the value it answers with is due for one. done is false only when +// the key is a miss and the caller still has time to wait for a load. +func (c *Cache[K, V]) cached(ctx context.Context, key K, loader func(context.Context, K) (V, error)) (v V, done bool, err error) { + v, st, _, refresh := c.core.lookup(key, c.core.refresherWith(loader)) + if refresh { + c.core.refresh(ctx, key, loader) + } + + switch st { case StatusHit: return v, true, nil case StatusNegative: @@ -133,7 +140,7 @@ func (c *core[K, V]) load(ctx context.Context, key K, loader func(context.Contex // Without it, a caller descheduled between the two would start a second load // for a value that is already cached. Counted without stats, because the // caller's lookup has already been counted as the miss it was. - if v, st, _ := s.get(key, c.now()); st != StatusMiss { + if v, st, _, _ := s.get(key, c.now(), refreshNone); st != StatusMiss { s.endLoad(key, token) g.mu.Unlock() c.countCoalesced(s) @@ -154,11 +161,37 @@ func (c *core[K, V]) load(ctx context.Context, key K, loader func(context.Contex g.calls[key] = cl g.mu.Unlock() - go c.run(loadCtx, g, s, key, cl, loader) + go c.run(loadCtx, g, s, key, cl, loader, false) return g.wait(ctx, key, cl) } +// refresh reloads key in the background ahead of its expiry. Nobody waits for +// it, but it is an ordinary flight: a caller that finds the key expired before +// it finishes joins it, and invalidation keeps its result out. A load for the +// key that is already running is left to publish instead. +func (c *core[K, V]) refresh(ctx context.Context, key K, loader func(context.Context, K) (V, error)) { + idx := c.shardIndex(key) + g, s := c.flights[idx], c.shards[idx] + + g.mu.Lock() + if cl, ok := g.calls[key]; ok && cl.token.valid.Load() { + g.mu.Unlock() + + return + } + + // The refresh is a waiter of its own that never leaves, so that a caller + // who joins it and gives up cannot cancel a load the cache asked for. + token := s.beginLoad(key) + loadCtx, cancel := context.WithCancel(context.WithoutCancel(ctx)) + cl := &call[V]{done: make(chan struct{}), waiters: 1, cancel: cancel, token: token} + g.calls[key] = cl + g.mu.Unlock() + + go c.run(loadCtx, g, s, key, cl, loader, true) +} + func (c *core[K, V]) countCoalesced(s *shard[K, V]) { if c.countStats { s.counters.coalesced.Add(1) @@ -166,16 +199,18 @@ func (c *core[K, V]) countCoalesced(s *shard[K, V]) { } // run performs the load and publishes the result. It runs on its own goroutine -// so that a caller can walk away from a load without ending it. +// so that a caller can walk away from a load without ending it. refresh says +// the load is a refresh, which is counted apart from loads. func (c *core[K, V]) run( - ctx context.Context, g *flightGroup[K, V], s *shard[K, V], key K, cl *call[V], loader func(context.Context, K) (V, error), + ctx context.Context, g *flightGroup[K, V], s *shard[K, V], key K, cl *call[V], + loader func(context.Context, K) (V, error), refresh bool, ) { defer s.endLoad(key, cl.token) g.run(key, cl, func() (V, error) { v, err := loader(ctx, key) switch { case err == nil: - _ = c.setValue(key, v, c.valueCost(v), c.ttl, cl.token) + _ = c.setValue(key, v, c.valueCost(v), c.ttl, c.refreshAfter, refreshCache, cl.token) case errors.Is(err, ErrNotFound): _ = c.setNegative(key, c.negativeTTL, cl.token) } @@ -183,7 +218,7 @@ func (c *core[K, V]) run( return v, err }, func(o loadOutcome) { if c.countStats { - s.counters.countLoad(o) + s.counters.countLoad(o, refresh) } }) } diff --git a/loader_batch.go b/loader_batch.go index d5a4e6a..4e8045d 100644 --- a/loader_batch.go +++ b/loader_batch.go @@ -33,6 +33,10 @@ import ( // // Without a BatchLoader the missing keys are loaded concurrently with // Options.Loader, one call per key. With neither, it returns ErrNoLoader. +// +// With RefreshAfter, the hits due for a refresh are refreshed together in one +// BatchLoader call of their own, apart from the missing keys: nobody waits for +// them, so they should not hold up the batch somebody does wait for. func (c *Cache[K, V]) GetManyOrLoad(ctx context.Context, keys []K) (map[K]V, error) { cr := c.core if cr.loader == nil { @@ -43,8 +47,9 @@ func (c *Cache[K, V]) GetManyOrLoad(ctx context.Context, keys []K) (map[K]V, err // duplicates; the set of keys that were not hits is only built once there is // one, so that the common case costs one map rather than two. found := make(map[K]V, len(keys)) - var missing []K + var missing, due []K var notHit map[K]struct{} + by := cr.refresherWith(cr.loader) for _, key := range keys { if _, dup := found[key]; dup { continue @@ -53,7 +58,10 @@ func (c *Cache[K, V]) GetManyOrLoad(ctx context.Context, keys []K) (map[K]V, err continue } - v, st := c.Lookup(key) + v, st, _, refresh := cr.lookup(key, by) + if refresh { + due = append(due, key) + } if st == StatusHit { found[key] = v @@ -67,6 +75,9 @@ func (c *Cache[K, V]) GetManyOrLoad(ctx context.Context, keys []K) (map[K]V, err missing = append(missing, key) } } + if len(due) > 0 { + cr.refreshMany(ctx, due) + } if len(missing) == 0 { return found, nil } @@ -77,10 +88,10 @@ func (c *Cache[K, V]) GetManyOrLoad(ctx context.Context, keys []K) (map[K]V, err waits, fresh, batch := cr.acquire(ctx, missing, found) switch { case batch != nil: - go cr.runBatch(batch.ctx, batch.cancel, fresh) + go cr.runBatch(batch.ctx, batch.cancel, fresh, false) default: for _, f := range fresh { - go cr.run(f.ctx, f.g, f.s, f.key, f.cl, cr.loader) + go cr.run(f.ctx, f.g, f.s, f.key, f.cl, cr.loader, false) } } @@ -147,7 +158,7 @@ func (c *core[K, V]) acquire(ctx context.Context, keys []K, found map[K]V) (wait // The same recheck as load's, for the same reason: without it a caller // descheduled after its lookup would load a value already cached. token := s.beginLoad(key) - if v, st, _ := s.get(key, c.now()); st != StatusMiss { + if v, st, _, _ := s.get(key, c.now(), refreshNone); st != StatusMiss { s.endLoad(key, token) g.mu.Unlock() c.countCoalesced(s) @@ -182,9 +193,49 @@ func (c *core[K, V]) acquire(ctx context.Context, keys []K, found map[K]V) (wait return waits, fresh, batch } +// refreshMany is refresh for several keys, in one BatchLoader call when there is +// a batch loader. Keys whose load is already running are left to it. +func (c *core[K, V]) refreshMany(ctx context.Context, keys []K) { + if c.batchLoader == nil { + for _, key := range keys { + c.refresh(ctx, key, c.loader) + } + + return + } + + var fresh []flight[K, V] + batch := &batchLoad{} + batch.ctx, batch.cancel = context.WithCancel(context.WithoutCancel(ctx)) + for _, key := range keys { + idx := c.shardIndex(key) + g, s := c.flights[idx], c.shards[idx] + + g.mu.Lock() + if cl, ok := g.calls[key]; ok && cl.token.valid.Load() { + g.mu.Unlock() + + continue + } + // One waiter that never leaves, as in refresh. + cl := &call[V]{done: make(chan struct{}), waiters: 1, token: s.beginLoad(key), cancel: batch.join()} + g.calls[key] = cl + g.mu.Unlock() + + fresh = append(fresh, flight[K, V]{key: key, g: g, s: s, cl: cl, ctx: batch.ctx}) + } + if len(fresh) == 0 { + batch.cancel() + + return + } + + go c.runBatch(batch.ctx, batch.cancel, fresh, true) +} + // runBatch performs one batch load and publishes each key's result, the way run -// does for one key. -func (c *core[K, V]) runBatch(ctx context.Context, cancel context.CancelFunc, fresh []flight[K, V]) { +// does for one key. refresh says the batch is a refresh. +func (c *core[K, V]) runBatch(ctx context.Context, cancel context.CancelFunc, fresh []flight[K, V], refresh bool) { defer cancel() defer func() { for _, f := range fresh { @@ -205,7 +256,7 @@ func (c *core[K, V]) runBatch(ctx context.Context, cancel context.CancelFunc, fr if c.countStats { fresh[0].s.counters.batches.Add(1) for _, f := range fresh { - f.s.counters.countLoad(outcomeOf(f.cl.err, f.cl.pan)) + f.s.counters.countLoad(outcomeOf(f.cl.err, f.cl.pan), refresh) } } for _, f := range fresh { @@ -232,7 +283,7 @@ func (c *core[K, V]) runBatch(ctx context.Context, cancel context.CancelFunc, fr } case ok: f.cl.val = v - _ = c.setValue(f.key, v, c.valueCost(v), c.ttl, f.cl.token) + _ = c.setValue(f.key, v, c.valueCost(v), c.ttl, c.refreshAfter, refreshCache, f.cl.token) default: f.cl.err = ErrNotFound _ = c.setNegative(f.key, c.negativeTTL, f.cl.token) diff --git a/refresh_test.go b/refresh_test.go new file mode 100644 index 0000000..e80fa8f --- /dev/null +++ b/refresh_test.go @@ -0,0 +1,611 @@ +package sanecache + +import ( + "context" + "errors" + "fmt" + "slices" + "strings" + "sync" + "sync/atomic" + "testing" + "time" +) + +const ( + refreshTTL = 10 * time.Minute + refreshAfter = 5 * time.Minute +) + +// manualClock makes the cache read its time from a counter the test moves, so +// that "the value is now due" is a step in the test rather than a sleep. It must +// be installed before the first write. +func manualClock[K comparable, V any](c *Cache[K, V]) (advance func(time.Duration)) { + c.core.coarse = new(atomic.Int64) + c.core.coarse.Store(time.Now().UnixNano()) + + return func(d time.Duration) { c.core.coarse.Add(int64(d)) } +} + +// flying reports whether a load for key is registered, which refresh does +// before it returns: a test can tell "no refresh was started" without waiting. +func flying[V any](c *Cache[string, V], key string) bool { + g := c.core.flights[c.core.shardIndex(key)] + g.mu.Lock() + defer g.mu.Unlock() + _, ok := g.calls[key] + + return ok +} + +// versionLoader returns 1, 2, 3… on successive calls. Calls after the first park +// until gate is closed, so that a test can look at the cache while a refresh is +// still running. +type versionLoader struct { + calls atomic.Int64 + gate chan struct{} +} + +func newVersionLoader() *versionLoader { return &versionLoader{gate: make(chan struct{})} } + +func (l *versionLoader) load(ctx context.Context, _ string) (int, error) { + n := l.calls.Add(1) + if n > 1 { + select { + case <-l.gate: + case <-ctx.Done(): + return 0, ctx.Err() + } + } + + return int(n), nil +} + +func newRefreshCache(t *testing.T, policy Policy, loader func(context.Context, string) (int, error)) (*Cache[string, int], func(time.Duration)) { + t.Helper() + c := New(Options[string, int]{ + TTL: refreshTTL, NegativeTTL: time.Minute, RefreshAfter: refreshAfter, + Policy: policy, Loader: loader, DisableCleanup: true, + }) + t.Cleanup(c.Close) + + return c, manualClock(c) +} + +// TestRefreshServesTheOldValueAndReloadsOnce is what RefreshAfter is for: a key +// in steady use is replaced in the background, its readers never wait for the +// upstream, and however many of them see it due, the upstream is asked once. +func TestRefreshServesTheOldValueAndReloadsOnce(t *testing.T) { + for _, policy := range []Policy{LRU, ClearOnFull} { + t.Run(policy.String(), func(t *testing.T) { + l := newVersionLoader() + c, advance := newRefreshCache(t, policy, l.load) + ctx := context.Background() + + if v, err := c.GetOrLoad(ctx, "k"); v != 1 || err != nil { + t.Fatalf("first load: %d, %v", v, err) + } + advance(refreshAfter + time.Second) + + var wg sync.WaitGroup + for range 50 { + wg.Add(1) + go func() { + defer wg.Done() + if v, err := c.GetOrLoad(ctx, "k"); v != 1 || err != nil { + t.Errorf("a due value is served while it refreshes: got %d, %v", v, err) + } + }() + } + wg.Wait() + + waitFor(t, "the refresh to start", func() bool { return l.calls.Load() == 2 }) + close(l.gate) + waitFor(t, "the refreshed value", func() bool { v, _ := c.Get("k"); return v == 2 }) + + if n := l.calls.Load(); n != 2 { + t.Fatalf("loader called %d times, want 2: one load and one refresh", n) + } + st := c.Stats() + if st.Loads != 1 || st.Refreshes != 1 || st.RefreshErrors != 0 || st.Misses != 1 || st.Expirations != 0 { + t.Fatalf("stats: %+v", st) + } + }) + } +} + +func TestRefreshWaitsUntilTheValueIsDue(t *testing.T) { + l := newVersionLoader() + c, advance := newRefreshCache(t, LRU, l.load) + ctx := context.Background() + + _, _ = c.GetOrLoad(ctx, "k") + advance(refreshAfter - time.Second) + if v, _ := c.GetOrLoad(ctx, "k"); v != 1 { + t.Fatalf("got %d", v) + } + if flying(c, "k") || l.calls.Load() != 1 { + t.Fatal("a value refreshed before RefreshAfter") + } +} + +// TestRefreshFailureKeepsTheValueUntilItExpires pins the failure policy: one +// attempt per value, the value served to its TTL, and the error reaching a +// caller only when the TTL is over, exactly as it would without refresh. +func TestRefreshFailureKeepsTheValueUntilItExpires(t *testing.T) { + var calls atomic.Int64 + c, advance := newRefreshCache(t, LRU, func(context.Context, string) (int, error) { + if calls.Add(1) == 1 { + return 1, nil + } + + return 0, errUpstream + }) + ctx := context.Background() + + _, _ = c.GetOrLoad(ctx, "k") + advance(refreshAfter + time.Second) + if v, err := c.GetOrLoad(ctx, "k"); v != 1 || err != nil { + t.Fatalf("got %d, %v", v, err) + } + waitFor(t, "the refresh to fail", func() bool { return c.Stats().RefreshErrors == 1 }) + + for range 5 { + if v, err := c.GetOrLoad(ctx, "k"); v != 1 || err != nil { + t.Fatalf("after a failed refresh: got %d, %v", v, err) + } + } + if flying(c, "k") || calls.Load() != 2 { + t.Fatalf("a failed refresh was retried: %d loader calls", calls.Load()) + } + + advance(refreshTTL) + if _, err := c.GetOrLoad(ctx, "k"); !errors.Is(err, errUpstream) { + t.Fatalf("past the TTL the caller loads and sees the error, got %v", err) + } + if st := c.Stats(); st.Refreshes != 1 || st.RefreshErrors != 1 || st.Loads != 2 || st.LoadErrors != 1 { + t.Fatalf("stats: %+v", st) + } +} + +func TestRefreshPanicIsCountedAndContained(t *testing.T) { + var calls atomic.Int64 + c, advance := newRefreshCache(t, LRU, func(context.Context, string) (int, error) { + if calls.Add(1) > 1 { + panic("boom") + } + + return 1, nil + }) + + _, _ = c.GetOrLoad(context.Background(), "k") + advance(refreshAfter + time.Second) + if v, _ := c.Get("k"); v != 1 { + t.Fatalf("got %d", v) + } + waitFor(t, "the refresh to fail", func() bool { return c.Stats().RefreshErrors == 1 }) + if v, ok := c.Get("k"); v != 1 || !ok { + t.Fatalf("the value did not survive a panicking refresh: %d, %v", v, ok) + } +} + +// TestRefreshKeepsOutOfInvalidatedKeys holds a refresh to the 0.4 guarantee: a +// Delete or a successful write while it runs keeps its result out. +func TestRefreshKeepsOutOfInvalidatedKeys(t *testing.T) { + for _, mutate := range []string{"delete", "set", "clear"} { + t.Run(mutate, func(t *testing.T) { + l := newVersionLoader() + c, advance := newRefreshCache(t, LRU, l.load) + + _, _ = c.GetOrLoad(context.Background(), "k") + advance(refreshAfter + time.Second) + _, _ = c.Get("k") + waitFor(t, "the refresh to start", func() bool { return l.calls.Load() == 2 }) + + switch mutate { + case "delete": + c.Delete("k") + case "set": + _ = c.Set("k", 100) + case "clear": + c.Clear() + } + close(l.gate) + waitFor(t, "the refresh to finish", func() bool { return c.Stats().Refreshes == 1 }) + + want, wantOK := 0, false + if mutate == "set" { + want, wantOK = 100, true + } + if v, ok := c.Get("k"); v != want || ok != wantOK { + t.Fatalf("got %d, %v; want %d, %v", v, ok, want, wantOK) + } + }) + } +} + +// TestExpiredKeyJoinsTheRunningRefresh: a refresh slower than what was left of +// the TTL must not be followed by a second load of the same key, and a caller +// who joins it and gives up must not cancel it. +func TestExpiredKeyJoinsTheRunningRefresh(t *testing.T) { + l := newVersionLoader() + c, advance := newRefreshCache(t, LRU, l.load) + + _, _ = c.GetOrLoad(context.Background(), "k") + advance(refreshAfter + time.Second) + _, _ = c.Get("k") + waitFor(t, "the refresh to start", func() bool { return l.calls.Load() == 2 }) + advance(refreshTTL) + + impatient, cancel := context.WithCancel(context.Background()) + gaveUp := make(chan error, 1) + go func() { + _, err := c.GetOrLoad(impatient, "k") + gaveUp <- err + }() + waitFor(t, "the caller to join the refresh", func() bool { return c.Stats().Coalesced == 1 }) + cancel() + if err := <-gaveUp; !errors.Is(err, context.Canceled) { + t.Fatalf("got %v", err) + } + + got := make(chan int, 1) + go func() { + v, _ := c.GetOrLoad(context.Background(), "k") + got <- v + }() + waitFor(t, "a second caller to join the refresh", func() bool { return c.Stats().Coalesced == 2 }) + close(l.gate) + if v := <-got; v != 2 { + t.Fatalf("the joined caller got %d, want the refreshed 2", v) + } + if n := l.calls.Load(); n != 2 { + t.Fatalf("loader called %d times, want 2", n) + } +} + +func TestRefreshNotFound(t *testing.T) { + for _, negativeTTL := range []time.Duration{time.Minute, 0} { + t.Run(fmt.Sprint(negativeTTL), func(t *testing.T) { + var calls atomic.Int64 + c := New(Options[string, int]{ + TTL: refreshTTL, NegativeTTL: negativeTTL, RefreshAfter: refreshAfter, DisableCleanup: true, + Loader: func(context.Context, string) (int, error) { + if calls.Add(1) == 1 { + return 1, nil + } + + return 0, ErrNotFound + }, + }) + defer c.Close() + advance := manualClock(c) + + _, _ = c.GetOrLoad(context.Background(), "k") + advance(refreshAfter + time.Second) + _, _ = c.Get("k") + waitFor(t, "the refresh to finish", func() bool { return c.Stats().Refreshes == 1 }) + + want, wantStatus := 1, StatusHit + if negativeTTL > 0 { + want, wantStatus = 0, StatusNegative + } + if v, st := c.Lookup("k"); v != want || st != wantStatus { + t.Fatalf("got %d, %v; want %d, %v", v, st, want, wantStatus) + } + if st := c.Stats(); st.RefreshErrors != 0 { + t.Fatalf("ErrNotFound counted as a refresh error: %+v", st) + } + }) + } +} + +// TestRefreshNeedsSomethingToLoadWith: a read with no loader leaves a due value +// for one that has one, rather than claiming a refresh it cannot run. +func TestRefreshNeedsSomethingToLoadWith(t *testing.T) { + c := New(Options[string, int]{TTL: refreshTTL, RefreshAfter: refreshAfter, DisableCleanup: true}) + defer c.Close() + advance := manualClock(c) + + var calls atomic.Int64 + load := func(context.Context) (int, error) { return int(calls.Add(1)), nil } + ctx := context.Background() + + _, _ = c.GetOrLoadFunc(ctx, "k", load) + _ = c.Set("plain", 7) + advance(refreshAfter + time.Second) + + if v, ok := c.Get("k"); v != 1 || !ok || flying(c, "k") { + t.Fatal("Get without Options.Loader started a refresh") + } + if v, _ := c.GetOrLoadFunc(ctx, "k", load); v != 1 { + t.Fatalf("got %d", v) + } + waitFor(t, "GetOrLoadFunc to refresh with its own function", func() bool { v, _ := c.Get("k"); return v == 2 }) + + if v, ok := c.Get("plain"); v != 7 || !ok || flying(c, "plain") { + t.Fatal("a value written with Set was refreshed without a loader") + } +} + +func TestGetRefreshesWithTheOptionsLoader(t *testing.T) { + l := newVersionLoader() + close(l.gate) + c, advance := newRefreshCache(t, LRU, l.load) + + _ = c.Set("k", 0) + advance(refreshAfter + time.Second) + if v, ok := c.Get("k"); v != 0 || !ok { + t.Fatalf("got %d, %v", v, ok) + } + waitFor(t, "Get to refresh with Options.Loader", func() bool { v, _ := c.Get("k"); return v == 1 }) +} + +// TestSetTTLShorterThanRefreshAfterIsNotRefreshed: a value whose own TTL leaves +// no room for a refresh before it is gone is not refreshed. +func TestSetTTLShorterThanRefreshAfterIsNotRefreshed(t *testing.T) { + l := newVersionLoader() + c, advance := newRefreshCache(t, LRU, l.load) + + _ = c.SetTTL("short", 0, refreshAfter) + _ = c.SetTTL("forever", 0, 0) + advance(refreshAfter - time.Second) + _, _ = c.Get("short") + advance(time.Hour) + _, _ = c.Get("forever") + if flying(c, "short") || flying(c, "forever") || l.calls.Load() != 0 { + t.Fatal("refreshed a value whose TTL has no room for it") + } +} + +func TestGetManyOrLoadRefreshesDueKeysInOneBatch(t *testing.T) { + rec := newBatchRecorder(map[string]int{"a": 1, "b": 2, "c": 3, "d": 4}, false) + c := New(Options[string, int]{ + TTL: refreshTTL, RefreshAfter: refreshAfter, BatchLoader: rec.load, DisableCleanup: true, + }) + defer c.Close() + advance := manualClock(c) + ctx := context.Background() + + if _, err := c.GetManyOrLoad(ctx, []string{"a", "b", "c"}); err != nil { + t.Fatal(err) + } + advance(refreshAfter + time.Second) + + got, err := c.GetManyOrLoad(ctx, []string{"a", "b", "c", "d", "a"}) + if err != nil || len(got) != 4 { + t.Fatalf("got %v, %v", got, err) + } + waitFor(t, "the refresh batch", func() bool { return c.Stats().Refreshes == 3 }) + + rec.mu.Lock() + batches := slices.Clone(rec.batches) + rec.mu.Unlock() + sorted := make([]string, 0, len(batches)) + for _, b := range batches { + b = slices.Clone(b) + slices.Sort(b) + sorted = append(sorted, strings.Join(b, ",")) + } + slices.Sort(sorted) + if want := []string{"a,b,c", "a,b,c", "d"}; !slices.Equal(sorted, want) { + t.Fatalf("batches %v, want %v", sorted, want) + } + if st := c.Stats(); st.Batches != 3 || st.Loads != 4 { + t.Fatalf("stats: %+v", st) + } +} + +// TestViewRefresh: a view refreshes its own values with its own loader, and the +// cache underneath neither refreshes them with its loader nor takes the refresh +// away from the view. +func TestViewRefresh(t *testing.T) { + var parentCalls atomic.Int64 + c := New(Options[string, any]{ + TTL: time.Hour, RefreshAfter: 30 * time.Minute, DisableCleanup: true, + Loader: func(context.Context, string) (any, error) { parentCalls.Add(1); return "parent", nil }, + }) + defer c.Close() + advance := manualClock(c) + + var viewCalls atomic.Int64 + v := NewView(c, ViewOptions[int]{ + Name: "n", TTL: refreshTTL, RefreshAfter: refreshAfter, + Loader: func(context.Context, string) (int, error) { return int(viewCalls.Add(1)), nil }, + }) + plain := NewView(c, ViewOptions[int]{Name: "plain"}) + + _, _ = v.GetOrLoad(context.Background(), "k") + _ = plain.Set("k", 5) + advance(refreshAfter + time.Second) + + if _, ok := c.Get("n:k"); !ok || flying(c, "n:k") || parentCalls.Load() != 0 { + t.Fatal("the cache refreshed a view's value with its own loader") + } + if got, _ := v.Get("k"); got != 1 { + t.Fatalf("got %d", got) + } + waitFor(t, "the view's refresh", func() bool { got, _ := v.Get("k"); return got == 2 }) + if st := v.Stats(); st.Refreshes != 1 || st.Loads != 1 { + t.Fatalf("view stats: %+v", st) + } + if st := c.Stats(); st.Refreshes != 1 { + t.Fatalf("cache stats: %+v", st) + } + + advance(refreshAfter + time.Second) + if got, _ := plain.Get("k"); got != 5 || parentCalls.Load() != 0 { + t.Fatal("a view without RefreshAfter was refreshed") + } +} + +func TestRefreshAfterValidation(t *testing.T) { + for name, o := range map[string]Options[string, int]{ + "negative": {TTL: time.Minute, RefreshAfter: -1}, + "without TTL": {RefreshAfter: time.Minute}, + "equal to TTL": {TTL: time.Minute, RefreshAfter: time.Minute}, + "longer than TTL": {TTL: time.Minute, RefreshAfter: time.Hour}, + "negative, with TTL": {TTL: time.Minute, RefreshAfter: -time.Second}, + } { + t.Run(name, func(t *testing.T) { + defer func() { + if recover() == nil { + t.Fatal("New accepted it") + } + }() + New(o) + }) + } + + c := New(Options[string, any]{TTL: time.Hour, DisableCleanup: true}) + defer c.Close() + for name, o := range map[string]ViewOptions[int]{ + "negative": {Name: "a", RefreshAfter: -1}, + "not shorter than own TTL": {Name: "b", TTL: time.Minute, RefreshAfter: time.Minute}, + "not shorter than cache's": {Name: "c", RefreshAfter: time.Hour}, + } { + t.Run("view "+name, func(t *testing.T) { + defer func() { + if recover() == nil { + t.Fatal("NewView accepted it") + } + }() + NewView(c, o) + }) + } + + // The view's TTL may come from the cache; RefreshAfter is checked against it. + NewView(c, ViewOptions[int]{Name: "inherits", RefreshAfter: time.Minute}) + // Without a cache nothing is stored, so there is nothing to refresh either. + NewView(nil, ViewOptions[int]{Name: "off", RefreshAfter: time.Minute}) +} + +// TestRefreshStaysInsideTheJitteredLife: with jitter, the refresh moves with the +// expiry it belongs to and never lands at or past it. +func TestRefreshStaysInsideTheJitteredLife(t *testing.T) { + c := New(Options[string, int]{TTL: refreshTTL, RefreshAfter: refreshAfter, Jitter: 100, DisableCleanup: true}) + defer c.Close() + manualClock(c) + now := c.core.now() + + for range 10_000 { + expiresAt, refreshAt := c.core.deadlines(refreshTTL, refreshAfter) + if refreshAt <= now || refreshAt >= expiresAt { + t.Fatalf("refresh at %d outside (%d, %d)", refreshAt-now, 0, expiresAt-now) + } + // Half way through the TTL, half way through the jittered life. + if life, at := expiresAt-now, refreshAt-now; at < life/2-1 || at > life/2+1 { + t.Fatalf("refresh at %d of a life of %d", at, life) + } + } +} + +func TestGetManyOrLoadRefreshesWithLoaderKeyByKey(t *testing.T) { + var calls atomic.Int64 + c := New(Options[string, int]{ + TTL: refreshTTL, RefreshAfter: refreshAfter, DisableCleanup: true, + Loader: func(context.Context, string) (int, error) { return int(calls.Add(1)), nil }, + }) + defer c.Close() + advance := manualClock(c) + ctx := context.Background() + + _, _ = c.GetManyOrLoad(ctx, []string{"a", "b"}) + advance(refreshAfter + time.Second) + _, _ = c.GetManyOrLoad(ctx, []string{"a", "b"}) + waitFor(t, "both refreshes", func() bool { return c.Stats().Refreshes == 2 }) + if st := c.Stats(); st.Batches != 0 || calls.Load() != 4 { + t.Fatalf("stats %+v after %d loader calls", st, calls.Load()) + } +} + +// TestRefreshLeavesARunningLoadToPublish: a load of the key that is already +// running will publish a newer value than a refresh started now could, so the +// refresh is not started. A test has to set that state up by hand: it only +// arises when a value comes due between a load's publication and its exit. +func TestRefreshLeavesARunningLoadToPublish(t *testing.T) { + never := func(context.Context, string) (int, error) { + t.Error("a refresh started next to a running load") + + return 0, nil + } + running := func(c *core[string, int], key string) *call[int] { + token := &loadToken{} + token.valid.Store(true) + cl := &call[int]{done: make(chan struct{}), token: token} + g := c.flights[c.shardIndex(key)] + g.mu.Lock() + g.calls[key] = cl + g.mu.Unlock() + + return cl + } + + c := New(Options[string, int]{TTL: refreshTTL, RefreshAfter: refreshAfter, Loader: never, DisableCleanup: true}) + defer c.Close() + cl := running(c.core, "k") + c.core.refresh(context.Background(), "k", never) + if g := c.core.flights[c.core.shardIndex("k")]; g.calls["k"] != cl { + t.Fatal("refresh replaced the running load") + } + + b := New(Options[string, int]{ + TTL: refreshTTL, RefreshAfter: refreshAfter, DisableCleanup: true, + BatchLoader: func(context.Context, []string) (map[string]int, error) { + t.Error("a batch refresh started for keys that are all loading") + + return nil, nil + }, + }) + defer b.Close() + running(b.core, "x") + running(b.core, "y") + b.core.refreshMany(context.Background(), []string{"x", "y"}) + + vc := New(Options[string, any]{TTL: refreshTTL, DisableCleanup: true}) + defer vc.Close() + v := NewView(vc, ViewOptions[int]{Name: "v", RefreshAfter: refreshAfter, Loader: never}) + g := v.flights[vc.core.shardIndex("v:k")] + token := &loadToken{} + token.valid.Store(true) + vcl := &call[int]{done: make(chan struct{}), token: token} + g.mu.Lock() + g.calls["k"] = vcl + g.mu.Unlock() + v.refresh(context.Background(), "k", never) + if g.calls["k"] != vcl { + t.Fatal("a view refresh replaced the running load") + } + + // Give a wrongly started refresh the chance to call never. + time.Sleep(10 * time.Millisecond) +} + +func TestViewRefreshFailureIsCountedForTheView(t *testing.T) { + c := New(Options[string, any]{TTL: refreshTTL, DisableCleanup: true}) + defer c.Close() + advance := manualClock(c) + + var calls atomic.Int64 + v := NewView(c, ViewOptions[int]{ + Name: "v", RefreshAfter: refreshAfter, + Loader: func(context.Context, string) (int, error) { + if calls.Add(1) == 1 { + return 1, nil + } + + return 0, errUpstream + }, + }) + + _, _ = v.GetOrLoad(context.Background(), "k") + advance(refreshAfter + time.Second) + _, _ = v.GetOrLoad(context.Background(), "k") + waitFor(t, "the view's refresh to fail", func() bool { return v.Stats().RefreshErrors == 1 }) + if got, ok := v.Get("k"); got != 1 || !ok { + t.Fatalf("got %d, %v", got, ok) + } + if st := c.Stats(); st.RefreshErrors != 1 { + t.Fatalf("cache stats: %+v", st) + } +} diff --git a/shard.go b/shard.go index be97e59..0bc3686 100644 --- a/shard.go +++ b/shard.go @@ -12,11 +12,44 @@ type entry[K comparable, V any] struct { value V cost int64 expiresAt int64 // unix nanoseconds; 0 means "never expires" - negative bool // upstream said this key does not exist + + // refreshAt is when a read able to load should start reloading the entry in + // the background, in unix nanoseconds; 0 means never. It is atomic because + // the read that claims the refresh may hold only the read lock, and claiming + // is a swap to 0: one refresh per stored value, however many reads see it due. + refreshAt atomic.Int64 + + negative bool // upstream said this key does not exist + // byView marks an entry a View wrote. Only a view refreshes it, with its own + // loader: the cache's loader would be handed the view's prefixed key. + byView bool prev, next *entry[K, V] // head of the list is the most recently used entry } +// refresher says which reader may claim an entry's refresh: nobody, the cache +// itself, or a view. It is also what a write records as the entry's owner. +type refresher uint8 + +const ( + refreshNone refresher = iota + refreshCache + refreshView +) + +// claimRefresh reports whether this read, by a reader that can refresh, is the +// one to refresh e. The caller holds the shard lock, read or write, and checks +// for refreshNone itself: reads that cannot refresh are most reads, and they +// should not pay for a call. +func (e *entry[K, V]) claimRefresh(by refresher, now int64) bool { + if e.byView != (by == refreshView) { + return false + } + at := e.refreshAt.Load() + + return at != 0 && now >= at && e.refreshAt.CompareAndSwap(at, 0) +} + func (e *entry[K, V]) expired(now int64) bool { return e.expiresAt != 0 && now >= e.expiresAt } @@ -66,10 +99,11 @@ func newShard[K comparable, V any](maxBytes int64, maxEntries int, policy Policy } } -// get returns the entry's value along with its status. The third result reports -// that the lookup landed on an entry that had already expired, so the caller can -// account for it separately from a plain miss. -func (s *shard[K, V]) get(key K, now int64) (V, Status, bool) { +// get returns the entry's value along with its status. expired reports that the +// lookup landed on an entry that had already expired, so the caller can account +// for it separately from a plain miss. refresh reports that the hit was due for a +// refresh and that this reader, being the kind given by by, claimed it. +func (s *shard[K, V]) get(key K, now int64, by refresher) (_ V, _ Status, expired, refresh bool) { var zero V // Under ClearOnFull nothing is reordered on access, so reads take only the @@ -80,7 +114,7 @@ func (s *shard[K, V]) get(key K, now int64) (V, Status, bool) { e, ok := s.items[key] if !ok { s.mu.RUnlock() - return zero, StatusMiss, false + return zero, StatusMiss, false, false } if e.expired(now) { s.mu.RUnlock() @@ -97,16 +131,19 @@ func (s *shard[K, V]) get(key K, now int64) (V, Status, bool) { } s.mu.Unlock() - return zero, StatusMiss, removed + return zero, StatusMiss, removed, false } status, value := StatusHit, e.value if e.negative { status, value = StatusNegative, zero } + if by != refreshNone { + refresh = e.claimRefresh(by, now) + } s.mu.RUnlock() - return value, status, false + return value, status, false, refresh } s.mu.Lock() @@ -114,19 +151,23 @@ func (s *shard[K, V]) get(key K, now int64) (V, Status, bool) { e, ok := s.items[key] if !ok { - return zero, StatusMiss, false + return zero, StatusMiss, false, false } if e.expired(now) { s.remove(e) - return zero, StatusMiss, true + return zero, StatusMiss, true, false } s.touch(e) if e.negative { - return zero, StatusNegative, false + return zero, StatusNegative, false, false + } + + if by != refreshNone { + refresh = e.claimRefresh(by, now) } - return e.value, StatusHit, false + return e.value, StatusHit, false, refresh } // set stores e, returning the entry it replaced (if any) and the entries evicted diff --git a/stats.go b/stats.go index 50432a0..a013296 100644 --- a/stats.go +++ b/stats.go @@ -32,6 +32,15 @@ type Stats struct { // after it published. Against Loads it says how much the single flight is // actually saving. Coalesced int64 + // Refreshes counts the reloads RefreshAfter started that finished, however + // they ended, a key at a time as Loads does. They are not in Loads: Loads is + // what misses cost the upstream, and Refreshes is what keeping hot keys warm + // costs it instead. + Refreshes int64 + // RefreshErrors counts the refreshes that failed or panicked. The value they + // meant to replace stays until its TTL runs out. A refresh that ended in + // ErrNotFound is not one of them. + RefreshErrors int64 Entries int // entries currently held, expired-but-not-yet-swept included Bytes int64 // sum of the costs of those entries @@ -63,10 +72,21 @@ func (c *counters) addTo(s *Stats) { s.LoadErrors += c.loadErrors.Load() s.Batches += c.batches.Load() s.Coalesced += c.coalesced.Load() + s.Refreshes += c.refreshes.Load() + s.RefreshErrors += c.refreshErrors.Load() } -// countLoad records one completed load. -func (c *counters) countLoad(o loadOutcome) { +// countLoad records one completed load, or refresh when it was one. +func (c *counters) countLoad(o loadOutcome, refresh bool) { + if refresh { + c.refreshes.Add(1) + if o == loadFailed { + c.refreshErrors.Add(1) + } + + return + } + c.loads.Add(1) switch o { case loadNotFound: @@ -77,17 +97,19 @@ func (c *counters) countLoad(o loadOutcome) { } type counters struct { - hits atomic.Int64 - misses atomic.Int64 - negatives atomic.Int64 - typeMisses atomic.Int64 - evictions atomic.Int64 - expirations atomic.Int64 - replacements atomic.Int64 - rejections atomic.Int64 - loads atomic.Int64 - loadNotFound atomic.Int64 - loadErrors atomic.Int64 - batches atomic.Int64 - coalesced atomic.Int64 + hits atomic.Int64 + misses atomic.Int64 + negatives atomic.Int64 + typeMisses atomic.Int64 + evictions atomic.Int64 + expirations atomic.Int64 + replacements atomic.Int64 + rejections atomic.Int64 + loads atomic.Int64 + loadNotFound atomic.Int64 + loadErrors atomic.Int64 + batches atomic.Int64 + coalesced atomic.Int64 + refreshes atomic.Int64 + refreshErrors atomic.Int64 } diff --git a/view.go b/view.go index 698fea8..dad4d1c 100644 --- a/view.go +++ b/view.go @@ -48,6 +48,14 @@ type ViewOptions[T any] struct { // takes the cache's NegativeTTL, and if that is unset too, SetNegative on // this view reports ErrNegativeDisabled. NegativeTTL time.Duration + + // RefreshAfter is Options.RefreshAfter for this view's values, refreshed with + // this view's loaders: Loader for GetOrLoad, Get and Lookup, the caller's + // function for GetOrLoadFunc. It must be shorter than the view's TTL. Unlike + // TTL it is not taken from the cache when zero, since the cache's would have + // to fit every view's TTL; zero means this view does not refresh, and the + // cache's RefreshAfter never touches a view's values. + RefreshAfter time.Duration } // View is a typed window onto a cache that holds values of several types under @@ -65,13 +73,18 @@ type ViewOptions[T any] struct { // While the name and key together fit in 32 bytes the compiler keeps that on the // stack; past it, every read allocates. type View[T any] struct { - cache *Cache[string, any] - name string - prefix string - cost func(T) int64 - ttl time.Duration - negativeTTL time.Duration - countStats bool + cache *Cache[string, any] + name string + prefix string + cost func(T) int64 + ttl time.Duration + negativeTTL time.Duration + refreshAfter time.Duration + countStats bool + + // getRefresher is who Get and Lookup read as: refreshView with RefreshAfter + // and a Loader, refreshNone otherwise. Worked out once, as the cache does. + getRefresher refresher loader func(context.Context, string) (T, error) flights []*flightGroup[string, T] @@ -97,6 +110,8 @@ func NewView[T any](c *Cache[string, any], o ViewOptions[T]) *View[T] { panic(fmt.Sprintf("sanecache: ViewOptions.Name must not contain %q, got %q", viewSeparator, o.Name)) case o.TTL < 0 || o.NegativeTTL < 0: panic("sanecache: ViewOptions TTL and NegativeTTL must not be negative") + case o.RefreshAfter < 0: + panic("sanecache: ViewOptions.RefreshAfter must not be negative") } if c == nil { @@ -129,6 +144,12 @@ func NewView[T any](c *Cache[string, any], o ViewOptions[T]) *View[T] { if v.negativeTTL == 0 { v.negativeTTL = c.core.negativeTTL } + if o.RefreshAfter > 0 && (v.ttl == 0 || o.RefreshAfter >= v.ttl) { + panic(fmt.Sprintf("sanecache: ViewOptions.RefreshAfter must be shorter than the view's TTL, got %v with TTL %v", + o.RefreshAfter, v.ttl)) + } + v.refreshAfter = o.RefreshAfter + v.getRefresher = v.refresherWith(o.Loader) // Made with or without a view loader: GetOrLoadFunc brings its own. v.flights = make([]*flightGroup[string, T], len(c.core.shards)) @@ -153,8 +174,22 @@ func (v *View[T]) Get(key string) (T, bool) { // Lookup returns the cached value and how the view answered. An entry holding // some other type is reported as a miss and counted as a TypeMiss: the value is -// unusable here, so the caller has to go to the upstream either way. +// unusable here, so the caller has to go to the upstream either way. With +// RefreshAfter and a Loader, a value due for a refresh starts one. func (v *View[T]) Lookup(key string) (T, Status) { + // background rather than context.Background(): the call is what would keep + // this from inlining into Get, and that costs every view read a call. + return v.lookup(background, key, v.getRefresher, v.loader) +} + +// background is the context of the refreshes Get and Lookup start, which have no +// caller's context to carry. +var background = context.Background() + +// lookup is Lookup as reader by, refreshing with loader. +func (v *View[T]) lookup( + ctx context.Context, key string, by refresher, loader func(context.Context, string) (T, error), +) (T, Status) { var zero T if v.cache == nil { @@ -163,7 +198,7 @@ func (v *View[T]) Lookup(key string) (T, Status) { return zero, StatusMiss } - raw, st, shard := v.cache.core.lookup(v.prefix + key) + raw, st, shard, refresh := v.cache.core.lookup(v.prefix+key, by) switch st { case StatusHit: val, ok := raw.(T) @@ -174,6 +209,9 @@ func (v *View[T]) Lookup(key string) (T, Status) { return zero, StatusMiss } v.count(&v.stats.hits) + if refresh { + v.refresh(ctx, key, loader) + } return val, StatusHit @@ -201,7 +239,7 @@ func (v *View[T]) SetTTL(key string, value T, ttl time.Duration) error { return ErrDisabled } - return v.cache.core.setValue(v.prefix+key, value, v.valueCost(value), ttl) + return v.cache.core.setValue(v.prefix+key, value, v.valueCost(value), ttl, v.refreshAfter, refreshView) } // SetNegative records that the upstream reports no such key, for the view's @@ -269,6 +307,15 @@ func (c *core[K, V]) viewCounters(name string) *viewCounters { return counters } +// refresherWith is who a read of this view is when it would refresh with loader. +func (v *View[T]) refresherWith(loader func(context.Context, string) (T, error)) refresher { + if v.refreshAfter > 0 && loader != nil { + return refreshView + } + + return refreshNone +} + func (v *View[T]) valueCost(value T) int64 { if v.cost != nil { return v.cost(value) @@ -283,7 +330,16 @@ func (v *View[T]) count(c *atomic.Int64) { } } -func (v *View[T]) countLoad(o loadOutcome) { +func (v *View[T]) countLoad(o loadOutcome, refresh bool) { + if refresh { + v.count(&v.stats.refreshes) + if o == loadFailed { + v.count(&v.stats.refreshErrors) + } + + return + } + v.count(&v.stats.loads) switch o { case loadNotFound: @@ -310,6 +366,9 @@ type ViewStats struct { LoadNotFound int64 // completed loader calls that returned ErrNotFound LoadErrors int64 // completed loader calls that failed otherwise, or panicked Coalesced int64 // calls spared a load by another caller + + Refreshes int64 // completed refreshes started by RefreshAfter + RefreshErrors int64 // refreshes that failed or panicked } // HitRate reports hits as a fraction of all lookups, counting a cached negative @@ -324,25 +383,29 @@ func (s ViewStats) HitRate() float64 { } type viewCounters struct { - hits atomic.Int64 - misses atomic.Int64 - negatives atomic.Int64 - typeMisses atomic.Int64 - loads atomic.Int64 - loadNotFound atomic.Int64 - loadErrors atomic.Int64 - coalesced atomic.Int64 + hits atomic.Int64 + misses atomic.Int64 + negatives atomic.Int64 + typeMisses atomic.Int64 + loads atomic.Int64 + loadNotFound atomic.Int64 + loadErrors atomic.Int64 + coalesced atomic.Int64 + refreshes atomic.Int64 + refreshErrors atomic.Int64 } func (c *viewCounters) snapshot() ViewStats { return ViewStats{ - Hits: c.hits.Load(), - Misses: c.misses.Load(), - Negatives: c.negatives.Load(), - TypeMisses: c.typeMisses.Load(), - Loads: c.loads.Load(), - LoadNotFound: c.loadNotFound.Load(), - LoadErrors: c.loadErrors.Load(), - Coalesced: c.coalesced.Load(), + Hits: c.hits.Load(), + Misses: c.misses.Load(), + Negatives: c.negatives.Load(), + TypeMisses: c.typeMisses.Load(), + Loads: c.loads.Load(), + LoadNotFound: c.loadNotFound.Load(), + LoadErrors: c.loadErrors.Load(), + Coalesced: c.coalesced.Load(), + Refreshes: c.refreshes.Load(), + RefreshErrors: c.refreshErrors.Load(), } } diff --git a/view_loader.go b/view_loader.go index 66f0614..05fa194 100644 --- a/view_loader.go +++ b/view_loader.go @@ -19,7 +19,7 @@ func (v *View[T]) GetOrLoad(ctx context.Context, key string) (T, error) { return zero, ErrNoLoader } - if val, done, err := v.cached(ctx, key); done { + if val, done, err := v.cached(ctx, key, v.loader); done { return val, err } @@ -41,16 +41,17 @@ func (v *View[T]) GetOrLoadFunc(ctx context.Context, key string, load func(conte return zero, ErrNoLoader } - if val, done, err := v.cached(ctx, key); done { + loader := func(ctx context.Context, _ string) (T, error) { return load(ctx) } + if val, done, err := v.cached(ctx, key, loader); done { return val, err } - return v.loadWith(ctx, key, func(ctx context.Context, _ string) (T, error) { return load(ctx) }) + return v.loadWith(ctx, key, loader) } // cached is Cache.cached for a view. -func (v *View[T]) cached(ctx context.Context, key string) (val T, done bool, err error) { - switch val, st := v.Lookup(key); st { +func (v *View[T]) cached(ctx context.Context, key string, loader func(context.Context, string) (T, error)) (val T, done bool, err error) { + switch val, st := v.lookup(ctx, key, v.refresherWith(loader), loader); st { case StatusHit: return val, true, nil case StatusNegative: @@ -91,7 +92,7 @@ func (v *View[T]) loadUncached(ctx context.Context, key string, loader func(cont go g.run(key, cl, func() (T, error) { return loader(loadCtx, key) - }, v.countLoad) + }, func(o loadOutcome) { v.countLoad(o, false) }) return g.wait(ctx, key, cl) } @@ -117,7 +118,7 @@ func (v *View[T]) load(ctx context.Context, key string, loader func(context.Cont token := s.beginLoad(fullKey) // Recheck without counting another lookup: a load may have published since // the caller's miss. A value of a different type still needs a typed load. - raw, st, _ := s.get(fullKey, c.now()) + raw, st, _, _ := s.get(fullKey, c.now(), refreshNone) val, typed := raw.(T) if st == StatusNegative || (st == StatusHit && typed) { s.endLoad(fullKey, token) @@ -136,25 +137,56 @@ func (v *View[T]) load(ctx context.Context, key string, loader func(context.Cont g.calls[key] = cl g.mu.Unlock() - go func() { - defer s.endLoad(fullKey, token) - g.run(key, cl, func() (T, error) { - value, err := loader(loadCtx, key) - switch { - case err == nil: - _ = c.setValue(fullKey, value, v.valueCost(value), v.ttl, token) - case errors.Is(err, ErrNotFound): - _ = c.setNegative(fullKey, v.negativeTTL, token) - } - - return value, err - }, func(o loadOutcome) { - v.countLoad(o) - if v.countStats { - s.counters.countLoad(o) - } - }) - }() + go v.run(loadCtx, g, s, key, cl, loader, false) return g.wait(ctx, key, cl) } + +// refresh is Cache's refresh for a view's key, with the view's loader. +func (v *View[T]) refresh(ctx context.Context, key string, loader func(context.Context, string) (T, error)) { + c := v.cache.core + fullKey := v.prefix + key + idx := c.shardIndex(fullKey) + g, s := v.flights[idx], c.shards[idx] + + g.mu.Lock() + if cl, ok := g.calls[key]; ok && cl.token.valid.Load() { + g.mu.Unlock() + + return + } + token := s.beginLoad(fullKey) + loadCtx, cancel := context.WithCancel(context.WithoutCancel(ctx)) + cl := &call[T]{done: make(chan struct{}), waiters: 1, cancel: cancel, token: token} + g.calls[key] = cl + g.mu.Unlock() + + go v.run(loadCtx, g, s, key, cl, loader, true) +} + +// run is Cache's run for a view's key: the result is stored under the prefixed +// key, with the view's cost and lifetimes, and counted for the view too. +func (v *View[T]) run( + ctx context.Context, g *flightGroup[string, T], s *shard[string, any], key string, cl *call[T], + loader func(context.Context, string) (T, error), refresh bool, +) { + c := v.cache.core + fullKey := v.prefix + key + defer s.endLoad(fullKey, cl.token) + g.run(key, cl, func() (T, error) { + value, err := loader(ctx, key) + switch { + case err == nil: + _ = c.setValue(fullKey, value, v.valueCost(value), v.ttl, v.refreshAfter, refreshView, cl.token) + case errors.Is(err, ErrNotFound): + _ = c.setNegative(fullKey, v.negativeTTL, cl.token) + } + + return value, err + }, func(o loadOutcome) { + v.countLoad(o, refresh) + if v.countStats { + s.counters.countLoad(o, refresh) + } + }) +} From d9b8988ba9a5ab293c3d576871025b79c5800349 Mon Sep 17 00:00:00 2001 From: Andrey Ryabokon Date: Fri, 9 Oct 2026 08:08:52 +0300 Subject: [PATCH 2/9] Share flight bookkeeping and counters between cache and view loads --- cache.go | 22 ++---------- loader.go | 91 +++++++++++++++++++++++++++++++------------------ loader_batch.go | 4 +-- view.go | 55 +++++++----------------------- view_loader.go | 52 +++++++++------------------- 5 files changed, 91 insertions(+), 133 deletions(-) diff --git a/cache.go b/cache.go index a680bc5..0a730fe 100644 --- a/cache.go +++ b/cache.go @@ -259,7 +259,7 @@ type core[K comparable, V any] struct { // share a namespace in storage, so they share counters too; the map only // grows with the number of distinct names, which is fixed by the program. viewsMu sync.Mutex - views map[string]*viewCounters + views map[string]*counters // coarse holds the time a background goroutine last read, in unix // nanoseconds, or nil when the cache reads the clock itself. Zero means the @@ -412,26 +412,10 @@ func (c *Cache[K, V]) Clear() { } // Len reports how many entries are held, including expired ones not yet swept. -func (c *Cache[K, V]) Len() int { - n := 0 - for _, s := range c.core.shards { - entries, _ := s.stats() - n += entries - } - - return n -} +func (c *Cache[K, V]) Len() int { return c.Stats().Entries } // Bytes reports the summed cost of the entries held. -func (c *Cache[K, V]) Bytes() int64 { - var total int64 - for _, s := range c.core.shards { - _, bytes := s.stats() - total += bytes - } - - return total -} +func (c *Cache[K, V]) Bytes() int64 { return c.Stats().Bytes } // Stats returns a snapshot of the counters. It walks every shard, so poll it on // a metrics interval rather than per request. diff --git a/loader.go b/loader.go index bb19982..494c610 100644 --- a/loader.go +++ b/loader.go @@ -68,26 +68,27 @@ func (c *Cache[K, V]) GetOrLoadFunc(ctx context.Context, key K, load func(contex // cached answers a load from the cache when it can, starting a refresh with // loader when the value it answers with is due for one. done is false only when // the key is a miss and the caller still has time to wait for a load. -func (c *Cache[K, V]) cached(ctx context.Context, key K, loader func(context.Context, K) (V, error)) (v V, done bool, err error) { +func (c *Cache[K, V]) cached(ctx context.Context, key K, loader func(context.Context, K) (V, error)) (V, bool, error) { v, st, _, refresh := c.core.lookup(key, c.core.refresherWith(loader)) if refresh { c.core.refresh(ctx, key, loader) } - switch st { - case StatusHit: + return answered(ctx, v, st) +} + +// answered is cached's verdict on a lookup. ctx is checked after the lookup: a +// cached answer is worth having even to a caller who has run out of time. +func answered[V any](ctx context.Context, v V, st Status) (V, bool, error) { + if st == StatusHit { return v, true, nil - case StatusNegative: - return v, true, ErrNotFound } - - // Checked after the lookup: a cached answer is worth having even to a caller - // who has already run out of time, and costs nothing to give. - if err := ctx.Err(); err != nil { - return v, true, err + if st == StatusNegative { + return v, true, ErrNotFound } + err := ctx.Err() - return v, false, nil + return v, err != nil, err } // call is one load in flight. Waiters read val and err only after done is @@ -118,12 +119,47 @@ func newFlightGroup[K comparable, V any]() *flightGroup[K, V] { return &flightGroup[K, V]{calls: make(map[K]*call[V])} } +// running returns key's load if one is in flight and not invalidated. The caller +// holds g.mu. +func (g *flightGroup[K, V]) running(key K) *call[V] { + cl := g.calls[key] + if cl == nil || (cl.token != nil && !cl.token.valid.Load()) { + return nil + } + + return cl +} + +// startAndUnlock registers a load of key with one waiter, unlocks g.mu and runs +// the load on a goroutine of its own. The load outlives the caller that starts +// it, so it takes the caller's values but not its cancellation: see +// call.waiters. +func (g *flightGroup[K, V]) startAndUnlock( + ctx context.Context, key K, token *loadToken, run func(context.Context, *call[V]), +) *call[V] { + loadCtx, cancel := context.WithCancel(context.WithoutCancel(ctx)) + cl := &call[V]{done: make(chan struct{}), waiters: 1, cancel: cancel, token: token} + g.calls[key] = cl + g.mu.Unlock() + go run(loadCtx, cl) + + return cl +} + +// drop takes cl out of the map unless a newer load has replaced it. The caller +// holds g.mu. +func (g *flightGroup[K, V]) drop(key K, cl *call[V]) { + if g.calls[key] == cl { + delete(g.calls, key) + } +} + func (c *core[K, V]) load(ctx context.Context, key K, loader func(context.Context, K) (V, error)) (V, error) { idx := c.shardIndex(key) g, s := c.flights[idx], c.shards[idx] g.mu.Lock() - if cl, ok := g.calls[key]; ok && cl.token.valid.Load() { + if cl := g.running(key); cl != nil { cl.waiters++ g.mu.Unlock() c.countCoalesced(s) @@ -154,14 +190,9 @@ func (c *core[K, V]) load(ctx context.Context, key K, loader func(context.Contex return v, nil } - // The load outlives the caller that starts it, so it does not inherit that - // caller's cancellation: values yes, deadline no. See call.waiters. - loadCtx, cancel := context.WithCancel(context.WithoutCancel(ctx)) - cl := &call[V]{done: make(chan struct{}), waiters: 1, cancel: cancel, token: token} - g.calls[key] = cl - g.mu.Unlock() - - go c.run(loadCtx, g, s, key, cl, loader, false) + cl := g.startAndUnlock(ctx, key, token, func(ctx context.Context, cl *call[V]) { + c.run(ctx, g, s, key, cl, loader, false) + }) return g.wait(ctx, key, cl) } @@ -175,7 +206,7 @@ func (c *core[K, V]) refresh(ctx context.Context, key K, loader func(context.Con g, s := c.flights[idx], c.shards[idx] g.mu.Lock() - if cl, ok := g.calls[key]; ok && cl.token.valid.Load() { + if g.running(key) != nil { g.mu.Unlock() return @@ -183,13 +214,9 @@ func (c *core[K, V]) refresh(ctx context.Context, key K, loader func(context.Con // The refresh is a waiter of its own that never leaves, so that a caller // who joins it and gives up cannot cancel a load the cache asked for. - token := s.beginLoad(key) - loadCtx, cancel := context.WithCancel(context.WithoutCancel(ctx)) - cl := &call[V]{done: make(chan struct{}), waiters: 1, cancel: cancel, token: token} - g.calls[key] = cl - g.mu.Unlock() - - go c.run(loadCtx, g, s, key, cl, loader, true) + g.startAndUnlock(ctx, key, s.beginLoad(key), func(ctx context.Context, cl *call[V]) { + c.run(ctx, g, s, key, cl, loader, true) + }) } func (c *core[K, V]) countCoalesced(s *shard[K, V]) { @@ -269,8 +296,8 @@ func (g *flightGroup[K, V]) leave(key K, cl *call[V]) { g.mu.Lock() cl.waiters-- last := cl.waiters == 0 - if last && g.calls[key] == cl { - delete(g.calls, key) + if last { + g.drop(key, cl) } g.mu.Unlock() @@ -284,9 +311,7 @@ func (g *flightGroup[K, V]) leave(key K, cl *call[V]) { // that has already been handed out. func (g *flightGroup[K, V]) finish(key K, cl *call[V]) { g.mu.Lock() - if g.calls[key] == cl { - delete(g.calls, key) - } + g.drop(key, cl) g.mu.Unlock() close(cl.done) diff --git a/loader_batch.go b/loader_batch.go index 4e8045d..409f28b 100644 --- a/loader_batch.go +++ b/loader_batch.go @@ -146,7 +146,7 @@ func (c *core[K, V]) acquire(ctx context.Context, keys []K, found map[K]V) (wait g, s := c.flights[idx], c.shards[idx] g.mu.Lock() - if cl, ok := g.calls[key]; ok && cl.token.valid.Load() { + if cl := g.running(key); cl != nil { cl.waiters++ g.mu.Unlock() c.countCoalesced(s) @@ -212,7 +212,7 @@ func (c *core[K, V]) refreshMany(ctx context.Context, keys []K) { g, s := c.flights[idx], c.shards[idx] g.mu.Lock() - if cl, ok := g.calls[key]; ok && cl.token.valid.Load() { + if g.running(key) != nil { g.mu.Unlock() continue diff --git a/view.go b/view.go index dad4d1c..0a0a83d 100644 --- a/view.go +++ b/view.go @@ -90,7 +90,7 @@ type View[T any] struct { flights []*flightGroup[string, T] // stats is shared by every view of this name on this cache. - stats *viewCounters + stats *counters } // NewView opens a view named o.Name onto c. It panics on a name that cannot keep @@ -120,7 +120,7 @@ func NewView[T any](c *Cache[string, any], o ViewOptions[T]) *View[T] { name: o.Name, prefix: o.Name + viewSeparator, countStats: true, - stats: new(viewCounters), + stats: new(counters), flights: []*flightGroup[string, T]{newFlightGroup[string, T]()}, } @@ -272,7 +272,7 @@ func (v *View[T]) Delete(key string) bool { // with the same name on the same cache share them, as they share the keys. The // cache's own Stats counts the same lookups across every view. func (v *View[T]) Stats() ViewStats { - return v.stats.snapshot() + return v.stats.viewStats() } // ViewStats returns a snapshot of the counters of every view opened on the @@ -284,27 +284,28 @@ func (c *Cache[K, V]) ViewStats() map[string]ViewStats { stats := make(map[string]ViewStats, len(c.core.views)) for name, counters := range c.core.views { - stats[name] = counters.snapshot() + stats[name] = counters.viewStats() } return stats } // viewCounters returns the counters for a view name, creating them on first use. -func (c *core[K, V]) viewCounters(name string) *viewCounters { +// A view uses only some of them. +func (c *core[K, V]) viewCounters(name string) *counters { c.viewsMu.Lock() defer c.viewsMu.Unlock() if c.views == nil { - c.views = make(map[string]*viewCounters) + c.views = make(map[string]*counters) } - counters, ok := c.views[name] + cs, ok := c.views[name] if !ok { - counters = new(viewCounters) - c.views[name] = counters + cs = new(counters) + c.views[name] = cs } - return counters + return cs } // refresherWith is who a read of this view is when it would refresh with loader. @@ -330,25 +331,6 @@ func (v *View[T]) count(c *atomic.Int64) { } } -func (v *View[T]) countLoad(o loadOutcome, refresh bool) { - if refresh { - v.count(&v.stats.refreshes) - if o == loadFailed { - v.count(&v.stats.refreshErrors) - } - - return - } - - v.count(&v.stats.loads) - switch o { - case loadNotFound: - v.count(&v.stats.loadNotFound) - case loadFailed: - v.count(&v.stats.loadErrors) - } -} - // ViewStats is a snapshot of the counters for one view name, cumulative since // the first view with that name was opened on the cache. type ViewStats struct { @@ -382,20 +364,7 @@ func (s ViewStats) HitRate() float64 { return float64(s.Hits+s.Negatives) / float64(total) } -type viewCounters struct { - hits atomic.Int64 - misses atomic.Int64 - negatives atomic.Int64 - typeMisses atomic.Int64 - loads atomic.Int64 - loadNotFound atomic.Int64 - loadErrors atomic.Int64 - coalesced atomic.Int64 - refreshes atomic.Int64 - refreshErrors atomic.Int64 -} - -func (c *viewCounters) snapshot() ViewStats { +func (c *counters) viewStats() ViewStats { return ViewStats{ Hits: c.hits.Load(), Misses: c.misses.Load(), diff --git a/view_loader.go b/view_loader.go index 05fa194..a49c6ff 100644 --- a/view_loader.go +++ b/view_loader.go @@ -50,18 +50,10 @@ func (v *View[T]) GetOrLoadFunc(ctx context.Context, key string, load func(conte } // cached is Cache.cached for a view. -func (v *View[T]) cached(ctx context.Context, key string, loader func(context.Context, string) (T, error)) (val T, done bool, err error) { - switch val, st := v.lookup(ctx, key, v.refresherWith(loader), loader); st { - case StatusHit: - return val, true, nil - case StatusNegative: - return val, true, ErrNotFound - } - if err := ctx.Err(); err != nil { - return val, true, err - } +func (v *View[T]) cached(ctx context.Context, key string, loader func(context.Context, string) (T, error)) (T, bool, error) { + val, st := v.lookup(ctx, key, v.refresherWith(loader), loader) - return val, false, nil + return answered(ctx, val, st) } func (v *View[T]) loadWith(ctx context.Context, key string, loader func(context.Context, string) (T, error)) (T, error) { @@ -78,21 +70,16 @@ func (v *View[T]) loadUncached(ctx context.Context, key string, loader func(cont g := v.flights[0] g.mu.Lock() - if cl, ok := g.calls[key]; ok { + if cl := g.running(key); cl != nil { cl.waiters++ g.mu.Unlock() v.count(&v.stats.coalesced) return g.wait(ctx, key, cl) } - loadCtx, cancel := context.WithCancel(context.WithoutCancel(ctx)) - cl := &call[T]{done: make(chan struct{}), waiters: 1, cancel: cancel} - g.calls[key] = cl - g.mu.Unlock() - - go g.run(key, cl, func() (T, error) { - return loader(loadCtx, key) - }, func(o loadOutcome) { v.countLoad(o, false) }) + cl := g.startAndUnlock(ctx, key, nil, func(ctx context.Context, cl *call[T]) { + g.run(key, cl, func() (T, error) { return loader(ctx, key) }, func(o loadOutcome) { v.stats.countLoad(o, false) }) + }) return g.wait(ctx, key, cl) } @@ -108,7 +95,7 @@ func (v *View[T]) load(ctx context.Context, key string, loader func(context.Cont } g.mu.Lock() - if cl, ok := g.calls[key]; ok && cl.token.valid.Load() { + if cl := g.running(key); cl != nil { cl.waiters++ g.mu.Unlock() coalesced() @@ -132,12 +119,9 @@ func (v *View[T]) load(ctx context.Context, key string, loader func(context.Cont return val, nil } - loadCtx, cancel := context.WithCancel(context.WithoutCancel(ctx)) - cl := &call[T]{done: make(chan struct{}), waiters: 1, cancel: cancel, token: token} - g.calls[key] = cl - g.mu.Unlock() - - go v.run(loadCtx, g, s, key, cl, loader, false) + cl := g.startAndUnlock(ctx, key, token, func(ctx context.Context, cl *call[T]) { + v.run(ctx, g, s, key, cl, loader, false) + }) return g.wait(ctx, key, cl) } @@ -150,18 +134,14 @@ func (v *View[T]) refresh(ctx context.Context, key string, loader func(context.C g, s := v.flights[idx], c.shards[idx] g.mu.Lock() - if cl, ok := g.calls[key]; ok && cl.token.valid.Load() { + if g.running(key) != nil { g.mu.Unlock() return } - token := s.beginLoad(fullKey) - loadCtx, cancel := context.WithCancel(context.WithoutCancel(ctx)) - cl := &call[T]{done: make(chan struct{}), waiters: 1, cancel: cancel, token: token} - g.calls[key] = cl - g.mu.Unlock() - - go v.run(loadCtx, g, s, key, cl, loader, true) + g.startAndUnlock(ctx, key, s.beginLoad(fullKey), func(ctx context.Context, cl *call[T]) { + v.run(ctx, g, s, key, cl, loader, true) + }) } // run is Cache's run for a view's key: the result is stored under the prefixed @@ -184,8 +164,8 @@ func (v *View[T]) run( return value, err }, func(o loadOutcome) { - v.countLoad(o, refresh) if v.countStats { + v.stats.countLoad(o, refresh) s.counters.countLoad(o, refresh) } }) From f021b60b72f5a9cea49a8825d7fe8d9468d7ea17 Mon Sep 17 00:00:00 2001 From: Andrey Ryabokon Date: Fri, 9 Oct 2026 08:16:20 +0300 Subject: [PATCH 3/9] Put the batch example's description back on the batch example --- example_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/example_test.go b/example_test.go index 75cddbc..5a44066 100644 --- a/example_test.go +++ b/example_test.go @@ -164,8 +164,6 @@ func ExampleCache_GetOrLoad() { // upstream calls: 2 } -// Fetching many keys from an upstream that answers many at once: one call for -// whatever the cache does not already hold. // A search page is cached under a key that only summarises the query. The request // itself travels with the call instead of being parsed back out of the key. func ExampleCache_GetOrLoadFunc() { @@ -203,6 +201,8 @@ func ExampleCache_GetOrLoadFunc() { // upstream calls: 1 } +// Fetching many keys from an upstream that answers many at once: one call for +// whatever the cache does not already hold. func ExampleCache_GetManyOrLoad() { var upstreamCalls atomic.Int64 From 4354a5dda65248be52b001e019e1869a5a897574 Mon Sep 17 00:00:00 2001 From: Andrey Ryabokon Date: Fri, 9 Oct 2026 08:16:31 +0300 Subject: [PATCH 4/9] Shorten comments and the README --- CHANGELOG.md | 14 +- README.md | 471 ++++++++++++++++-------------------------------- cache.go | 211 ++++++++-------------- errors.go | 27 +-- example_test.go | 5 +- loader.go | 109 ++++------- loader_batch.go | 92 +++------- shard.go | 77 +++----- stats.go | 36 ++-- view.go | 131 +++++--------- view_loader.go | 30 +-- 11 files changed, 406 insertions(+), 797 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5898f49..00d2f04 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,14 +9,12 @@ While the major version is 0, the public API may change in any release. ### Added - `Options.RefreshAfter` and `ViewOptions.RefreshAfter`: a value read after it is this old - is returned at once and reloaded in the background, so a key in steady use is replaced - before it expires instead of making the next caller wait for the upstream. `GetOrLoad`, - `GetManyOrLoad` and `GetOrLoadFunc` refresh with their loaders, `Get` and `Lookup` with - the loaders in the options. Each value gets one refresh; a failed one leaves it in place - until its TTL, and the TTL is never extended. `GetManyOrLoad` refreshes the keys it finds - due in one `BatchLoader` call. Off by default; it must be shorter than the TTL. -- `Stats.Refreshes`, `Stats.RefreshErrors` and the same in `ViewStats`. Refreshes are not - counted in `Loads`. + is returned at once and reloaded in the background, so a key in steady use does not go + cold. Any read with a loader refreshes, `Get` and `Lookup` included; `GetManyOrLoad` + refreshes its due keys in one `BatchLoader` call. One refresh per value; a failed one + leaves the value to its TTL, which is never extended. Off by default; it must be shorter + than the TTL. +- `Stats.Refreshes` and `Stats.RefreshErrors`, also in `ViewStats`, apart from `Loads`. ### Changed diff --git a/README.md b/README.md index abc01c1..7cfba60 100644 --- a/README.md +++ b/README.md @@ -6,13 +6,11 @@ A small in-memory cache for Go that aims to be **predictable before it is fast**. -There is no shortage of Go caches, and several of them are excellent. This one exists -because the failures that actually cost time in production were never about throughput: -a write that reports success and is dropped a moment later, a hit rate that quietly -collapses because every value is too big for the budget, a limit expressed in entries -when the thing you are protecting is a memory limit, a batch of keys that all expire in -the same millisecond and stampede the upstream together, and a cold key that a hundred -concurrent requests each fetch for themselves. +There are excellent Go caches already. This one exists because the failures that cost time +in production were never about throughput: a write that reports success and is dropped a +moment later, a hit rate that collapses because every value is too big for the budget, a +limit in entries when the thing to protect is memory, keys that all expire in the same +millisecond, and a cold key that a hundred concurrent requests each fetch for themselves. ```go import "github.com/andared/sanecache" @@ -50,24 +48,19 @@ func main() { } ``` -With Go 1.24 or newer: - ```sh go mod init example.com/cache-demo go get github.com/andared/sanecache@v0.7.0 go run . ``` -Output: `hello true`. - -The same program is in [`examples/basic`](examples/basic). A second -[runnable example](examples/README.md#caching-an-http-dependency) caches JSON responses from -a local HTTP server, including HTTP 404 responses as negative entries. Both run in CI. +Output: `hello true`. The same program is in [`examples/basic`](examples/basic), and +[a second example](examples/README.md#caching-an-http-dependency) caches JSON responses +from a local HTTP server, 404s included as negative entries. Both run in CI. ## Applying a byte budget -In an application with its own `Article` type and `fetch` function, account for each value's -cost and handle missing records explicitly: +With your own `Article` type and `fetch` function: ```go c := sanecache.New(sanecache.Options[string, *Article]{ @@ -102,40 +95,28 @@ if err := c.Set(id, article); err != nil { ## What "sane" means here -**Writes are synchronous.** A value is readable the moment `Set` returns. Caches that -buffer writes asynchronously are faster on paper and force `Wait()` calls into your tests -to paper over the gap — and once your tests need it, so does any code that writes a value -and reads it back in the same request. +**Writes are synchronous.** A value is readable the moment `Set` returns, so tests need no +`Wait()`, and neither does code that writes a value and reads it back. -**A value that does not fit is refused, not swallowed.** `Set` returns `ErrTooLarge` when -the value's cost exceeds its shard's budget. The alternative — accept the write, return -success, drop the entry during eviction — is invisible: the hit rate does not obviously -look wrong, and every request for those keys goes to the upstream forever. +**A value that does not fit is refused.** `Set` returns `ErrTooLarge` when the value costs +more than its shard's budget, instead of accepting it and evicting it unseen, which sends +every request for that key to the upstream forever. -**Budgets are in bytes.** A cap on the number of entries tells you nothing about memory -when entries are documents rather than integers: ten thousand entries is a rounding error -for `int` values and several gigabytes for HTML templates. `MaxBytes` requires a `Cost` -function; asking for a byte budget without one is a construction-time panic rather than a -budget that silently counts entries. +**Budgets are in bytes.** Ten thousand entries is nothing for `int` values and gigabytes for +HTML templates. `MaxBytes` without a `Cost` function panics at construction rather than +quietly counting entries. -**"It does not exist" is an answer.** `SetNegative` records that the upstream was asked and -said no. Without a first-class form for this, it ends up as a sentinel value smuggled -inside your value type, which does not survive the type parameter and tends to be charged -zero cost — making it the one thing eviction can never reclaim. Under a byte budget a -negative entry is charged what it holds: the entry, its map slot and its key. +**"It does not exist" is an answer.** `SetNegative` records that the upstream said no, +instead of a sentinel smuggled into the value type. Under a byte budget a negative entry is +charged what it holds — entry, map slot and key — so eviction can reclaim it. -**TTLs can be jittered.** `Jitter: 10` spreads each expiry by up to ±10%. Keys warmed -together by one request otherwise expire together and hit the upstream as one wave. +**TTLs can be jittered.** `Jitter: 10` spreads each expiry by up to ±10%, so keys warmed +together do not expire together and hit the upstream as one wave. -**A cold key is fetched once.** `GetOrLoad` runs the loader once per key however many -callers arrive while it is running. +**A cold key is fetched once**, however many callers ask for it at the same moment. ## Loading a cold key once -The byte-budget example above — look up, go to the upstream on a miss, remember the answer, -remember the absence of one — is the same in every service, and it has a hole in it: on a -cold key, every concurrent request runs it at the same time. - ```go c := sanecache.New(sanecache.Options[string, *Article]{ TTL: 10 * time.Minute, @@ -158,42 +139,29 @@ case err != nil: } ``` -Three details worth knowing, because they are where implementations differ: - -**"Does not exist" is one error, wherever it came from.** The loader returns `ErrNotFound`; -`GetOrLoad` caches that as a negative entry and reports the same `ErrNotFound` to later -callers. Handling of "no such object" does not depend on whether the cache happened to -remember it. Translate the upstream's own error once, in the loader, and it stops mattering -anywhere else. - -**Giving up does not cancel the load for everyone else.** The loader's context is not the -first caller's. It carries that caller's values, but it is cancelled only when *every* -caller waiting on the result has gone. A bare -[`singleflight.Group`](https://pkg.go.dev/golang.org/x/sync/singleflight#Group) suppresses -duplicate calls but takes no context argument and defines no cancellation policy. If its -callback captures the first caller's context, that caller's timeout can cancel work that -other callers still need; handling this is the application's responsibility. A caller who -gives up here gets `ctx.Err()` and leaves the others alone. - -**Failures are not cached.** Anything other than `ErrNotFound` is passed back unchanged and -nothing is stored, so the next call tries again. Single-flight already collapses the retry -storm; caching the error on top of that would turn a blip into an outage that outlives it. - -The loader runs on a goroutine of the cache's own, which is what makes the two points above -possible. If it panics, the panic is carried to the callers rather than taking the process -down, and it arrives with the stack of where it actually happened. A load that needs a -deadline of its own should set one inside the loader, where the right number is known. - -A load with no caller left is cancelled. A loader that ignores cancellation can still -warm the cache, provided its result has not been superseded by an explicit invalidation -or a successful write, including another loader's publication. +`GetOrLoad` runs the loader once per key however many callers arrive while it runs. Where +implementations differ: + +- **"Does not exist" is one error.** The loader returns `ErrNotFound`, `GetOrLoad` caches it + as a negative entry and returns the same error to later callers. Translate the upstream's + own error once, in the loader. +- **Giving up does not cancel the load for everyone else.** The loader's context carries the + first caller's values but is cancelled only once *every* waiting caller has gone. A bare + [`singleflight.Group`](https://pkg.go.dev/golang.org/x/sync/singleflight#Group) leaves + that to you, and a callback capturing the first caller's context lets its timeout cancel + work others still need. +- **Failures are not cached.** Errors other than `ErrNotFound` are returned unchanged and the + next call tries again; single flight already collapses the retry storm. + +The loader runs on a goroutine of the cache's own. A panic in it reaches the callers, with +the stack of where it happened, rather than taking the process down. Set a deadline inside +the loader, where the right number is known. A loader that ignores cancellation still warms +the cache, unless the key was invalidated or written in the meantime. ### When the loader needs more than the key -A loader in `Options` gets the key and nothing else. When the upstream call needs more — a -query string the key only summarises, a request the caller has already built — the choice -is between encoding all of it into the key and parsing it back out in the loader, or -passing the loader in with the call: +When the upstream call needs more than the key — a query string the key only summarises, a +request already built — pass the loader in with the call: ```go page, err := c.GetOrLoadFunc(ctx, pageKey(path, params), func(ctx context.Context) (*Page, error) { @@ -201,39 +169,25 @@ page, err := c.GetOrLoadFunc(ctx, pageKey(path, params), func(ctx context.Contex }) ``` -It is `GetOrLoad` in every other respect, and shares loads with it: callers of one key wait -for whichever load started first, so the function must return what any other caller's would -for that key. Views have `GetOrLoadFunc` too. Neither needs a loader in the options. +It shares loads with `GetOrLoad`: callers of a key wait for whichever load started first, +so the function must return what any caller's would for that key. Views have it too. ## Invalidation while loading -`Delete` invalidates outstanding loads for the key, even when it returns `false` because -there was no stored entry. `Clear` invalidates outstanding loads as it clears each shard. -Successful `Set`, `SetTTL`, `SetNegative` and `SetNegativeTTL` calls also prevent older -loads from overwriting their result. Rejected writes leave outstanding loads unchanged. - -For example, if a loader reads an old record, and the application updates the source and -then calls `Delete`, that loader cannot put the old record back into the cache. Callers -already waiting for it still receive its result; invalidation neither cancels those -requests nor retries them. A new caller after invalidation does not join the obsolete load. -This is a cache-publication guarantee, not a guarantee that every running request sees -fresh data. Update the source before invalidating the cache. - -The guarantee follows the storage key: it applies across views sharing a namespace and -to writes or deletes made through the parent cache. Successful loader publications also -supersede other outstanding loads for the same storage key. Invalidation tracking exists -only for active loads; deleted keys do not leave permanent metadata behind. +`Delete` (even of an absent key), `Clear` and successful writes keep outstanding loads of +the key from putting their result in the cache. So if a loader reads an old record and the +application then updates the source and calls `Delete`, the old record does not come back. +Callers already waiting still get the loader's result; new callers start a new load. Update +the source before invalidating the cache. -`Clear` visits shards individually, so it is not an atomic snapshot or a pause on traffic. -New loads and writes may repopulate a shard after it has been cleared. `Close` still only -stops maintenance goroutines; it does not invalidate entries or cancel loaders. +This follows the storage key, across views sharing a namespace and the parent cache, and a +load's own publication supersedes other loads of the key. `Clear` works shard by shard, so +it is not an atomic snapshot. `Close` only stops background goroutines. ## Loading many keys at once -When the upstream answers many keys in one call — a query with `IN`, a pipelined round -trip — loading them one `GetOrLoad` at a time turns one request into as many upstream -calls as there are keys. `GetManyOrLoad` looks every key up and sends the ones the cache -does not hold to a `BatchLoader` in one call: +`GetManyOrLoad` sends the keys the cache does not hold to a `BatchLoader` in one call — a +query with `IN`, a pipelined round trip: ```go c := sanecache.New(sanecache.Options[int, *Article]{ @@ -247,43 +201,20 @@ c := sanecache.New(sanecache.Options[int, *Article]{ articles, err := c.GetManyOrLoad(ctx, ids) // cached or loaded; absent ids are not in it ``` -It keeps the guarantees of `GetOrLoad`, key by key rather than batch by batch: +Everything works per key: a key another call is already loading is waited for, not loaded +again, and a `Delete` of one key keeps only that key's result out. A key missing from the +answer is cached as a negative entry and is simply absent from the result. An error fails +the whole batch and nothing from it is cached. The upstream call is cancelled only once no +caller waits for any of its keys. -**Single flight covers each key.** A key that another `GetOrLoad` or `GetManyOrLoad` is -already loading is waited for, not loaded again, so overlapping batches send the upstream -only the keys nobody has asked for yet. A `GetOrLoad` for a key that is part of a running -batch waits for the batch. - -**"Does not exist" is a key missing from the answer.** It is cached as a negative entry, -as `ErrNotFound` from a `Loader` would be, and it is not an error: the key is simply absent -from the result. - -**A failed batch fails whole.** An error from the loader is returned for every key it was -asked for and nothing from that call is cached, including whatever the map held alongside -the error. Keys that were already cached, or that other loads supplied, are still returned -with it. - -**The upstream call lives as long as somebody wants any of it.** Its context is cancelled -once no caller is waiting for any of its keys, so a caller who gives up on part of a batch -does not cancel it for another caller who needs a different part. - -**Invalidation is per key.** A `Delete` or a successful write of one key during the batch -keeps that key's result out of the cache and leaves the rest of the batch alone. - -With only a `BatchLoader`, `GetOrLoad` uses it with a batch of one, so a cache needs a -single loader for both. With only a `Loader`, `GetManyOrLoad` loads the missing keys -concurrently, one call each. `Stats().Batches` counts `BatchLoader` calls; against -`Loads`, which counts keys, it says how many keys an upstream call carries. Batch loading -is not available on views yet. +With only a `BatchLoader`, `GetOrLoad` uses it with a batch of one; with only a `Loader`, +`GetManyOrLoad` loads the missing keys concurrently. `Stats().Batches` counts batch calls. +Views have no batch loading yet. ## Refreshing before expiry -A TTL is a promise about staleness, and a cache keeps it by going back to the upstream. On -a key in steady use it does that on the request path: the value expires, the next caller -misses and waits, and every key the service relies on goes cold once per TTL. In a busy -cache with a high hit rate, almost every miss can be exactly that — `Expirations` and -`Loads` moving in step is the sign of it. - +On a key in steady use, a TTL means the value expires, the next caller misses and waits, and +every hot key goes cold once per TTL. `Expirations` and `Loads` moving in step is the sign. `RefreshAfter` moves that load off the request path: ```go @@ -294,38 +225,21 @@ c := sanecache.New(sanecache.Options[int, *Rule]{ }) ``` -A read that finds a value older than `RefreshAfter` returns it at once and starts a load -in the background. The value in the cache is replaced when the load succeeds, and the TTL -starts over. A key nobody reads is not refreshed: it expires as it did before, so refresh -spends upstream calls only on keys that are still wanted. - -**One refresh per value.** However many reads see a value due, it gets one load. The -refresh takes part in single flight like any load: a caller that finds the key expired -before the refresh finishes waits for it instead of starting another, and a `Delete` or a -successful write keeps its result out. - -**The TTL still holds.** A failed refresh leaves the value in place until its TTL runs out, -and is not retried; after that the next caller loads the key and sees the error, as it -would without refresh. An upstream that is down therefore shows up one TTL later than it -otherwise would, and never later than that. A refresh answered with `ErrNotFound` turns the -value into a negative entry when `NegativeTTL` is set. - -**Any read with a loader refreshes.** `GetOrLoad` and `GetManyOrLoad` use the loaders in -`Options`, `GetOrLoadFunc` the function it was given, and `Get` and `Lookup` the loaders in -`Options` when there are any. `GetManyOrLoad` sends the keys it finds due to the -`BatchLoader` in one call of their own, apart from the keys it is waiting for. Views take a -`RefreshAfter` of their own and refresh with their own loaders. +A read that finds a value older than `RefreshAfter` returns it at once and starts one load +in the background; a key nobody reads simply expires. The refresh is an ordinary load to +single flight and invalidation. The TTL still holds: a failed refresh is not retried, the +value lives out its TTL, and the next caller then sees the error, so a dead upstream shows +up one TTL later than without refresh, never more. -Jitter moves the refresh along with the expiry it belongs to. `Stats().Refreshes` counts -finished refreshes and `RefreshErrors` the failed ones; neither is part of `Loads`, which -stays what misses cost. +`GetOrLoad`, `GetManyOrLoad` and `GetOrLoadFunc` refresh with their loaders, `Get` and +`Lookup` with the ones in `Options`. `GetManyOrLoad` refreshes due keys in a batch of their +own. Views take their own `RefreshAfter`. `Stats().Refreshes` and `RefreshErrors` count +refreshes, apart from `Loads`. ## Several value types under one budget -A budget in bytes is only worth having if it covers everything competing for the memory. -One cache per type means one budget per type, and dividing a fixed amount of memory between -types up front is exactly the guess the byte budget was meant to avoid: the split that was -right at deploy time is wrong by the next traffic pattern. +One cache per type means one budget per type, and splitting memory between types up front +is the guess a byte budget was meant to avoid. ```go c := sanecache.New(sanecache.Options[string, any]{ @@ -347,22 +261,12 @@ seasons := sanecache.NewView(c, sanecache.ViewOptions[*Season]{ a, ok := articles.Get(id) // a is a *Article, not an any ``` -A view fixes one value type, counts its own hits and misses, and prefixes its keys with its -name, so two views cannot collide on the same id. It can bring its own `Cost` — which is -what spares you the type switch that a shared `Cost func(any) int64` otherwise becomes — and -its own TTLs. Eviction stays global: a view that suddenly needs more memory takes it from -whichever entries were used least recently, whatever type they are. +A view fixes one value type, prefixes its keys with its name and can bring its own `Cost` +and TTLs. Eviction stays global. Reads cost about 6 ns more than the cache underneath, and +allocate once name and key exceed 32 bytes. `Stats().TypeMisses` counts lookups that found +another type under a view's key: a bug detector for two views sharing a name. -A view is a function rather than a method on `Cache` because a method cannot introduce a -type parameter of its own. Reads cost about 6 ns more than the cache underneath, and no -allocation while the name and key together stay within 32 bytes: up to that length the -compiler keeps the joined key on the stack, and past it every read allocates. - -`Stats().TypeMisses` counts lookups that found some other type under a view's key. With -names in the keys that should never happen, which is the point: it is a bug detector for -two views sharing a name, or a write made straight to the underlying cache. - -Views can load values too: +Views load too: ```go articles := sanecache.NewView(c, sanecache.ViewOptions[*Article]{ @@ -374,24 +278,14 @@ articles := sanecache.NewView(c, sanecache.ViewOptions[*Article]{ a, err := articles.GetOrLoad(ctx, id) ``` -The loader receives the original key, without the namespace prefix. Results use the -view's cost and TTLs within the shared budget. Errors, cancellation, oversized values -and panics follow the same rules as `Cache.GetOrLoad` above; a wrong-type entry triggers -a typed load. With no view loader, `GetOrLoad` returns `ErrNoLoader` even for a cached key. - -Reuse the same `View` instance to share in-flight loads. Separate instances have -independent loaders and flights, including instances with the same name; their stored -entries still share that namespace. The underlying cache's loader is independent and is -never used as a fallback. `View.Stats()` includes `Loads`, `LoadNotFound`, `LoadErrors` and `Coalesced`; -the parent cache includes those events in its aggregate counters too. - -Counters belong to the view's name, not to the instance: views opened with the same name on -the same cache share one set, as they share one set of keys. +The loader gets the key without the prefix and follows the rules of `Cache.GetOrLoad`. +Loads are shared within one `View` instance; the cache's own loader is never a fallback. +Counters belong to the view's name and are included in the cache's. ### Switching caching off -A view opened on a nil cache stores nothing. It is for the configuration where a TTL of -zero means "do not cache", so that the code calling `GetOrLoad` stays the same either way: +A view opened on a nil cache stores nothing, so that code calling `GetOrLoad` stays the same +when a TTL of zero means "do not cache": ```go var cache *sanecache.Cache[string, any] @@ -405,19 +299,13 @@ articles := sanecache.NewView(cache, sanecache.ViewOptions[*Article]{ }) ``` -Such a view runs the loader on every call. Callers who arrive while a load is running still -share it — turning the cache off should not turn off the protection of the upstream — but -the result, `ErrNotFound` included, is not kept. Lookups are misses, `Delete` reports -`false`, and writes return `ErrDisabled` rather than a success the next read would -contradict. Its counters are its own and do not appear in any cache's `ViewStats`. +It runs the loader on every call, still shared by concurrent callers, and keeps nothing. +Writes return `ErrDisabled`. Its counters are its own. ## The `Cost` function is the part worth getting right -`MaxBytes` is only as honest as `Cost`. The number you want is resident size, and it is -usually several times the serialized size — a decoded struct carries headers, pointers, -map overhead and per-field padding that JSON does not. - -Measure it once instead of guessing: +`MaxBytes` is only as honest as `Cost`, and resident size is usually several times the +serialized size. Measure it once: ```go runtime.GC() @@ -437,19 +325,14 @@ runtime.KeepAlive(values) ratio := float64(after.HeapAlloc-before.HeapAlloc) / float64(n*len(sample)) ``` -In one production service this ratio came out at ~2.6× for decoded JSON structs, higher -for `map[string]any`, and ~6.7× for compiled templates. Sizing a cache off raw payload -length understated real memory sevenfold. +In one production service this came out at ~2.6× for decoded JSON structs, higher for +`map[string]any`, and ~6.7× for compiled templates. ## Sharding -`Shards` splits the cache into independently locked parts (rounded up to a power of two). -It is off by default, because it is not free: each shard gets an equal slice of `MaxBytes`, -so an uneven key distribution leaves part of the budget unused, and eviction becomes -per-shard rather than global. - -Whether it is worth it depends entirely on contention. On an M3 with GOMAXPROCS=8, 4096 -keys, all readers hitting one cache: +`Shards` splits the cache into independently locked parts, rounded up to a power of two. +It is off by default: each shard gets an equal slice of the budget, and eviction becomes +per shard. On an M3 with GOMAXPROCS=8, 4096 keys, all readers on one cache: | shards | `Get` (LRU) | `Get` (ClearOnFull) | `Set` | 90/10 mixed | |-------:|------------:|--------------------:|------:|------------:| @@ -458,29 +341,19 @@ keys, all readers hitting one cache: | 16 | 40 ns | 24 ns | 76 ns | 47 ns | | 64 | 33 ns | 19 ns | 65 ns | 39 ns | -Sharding is worth about 4x here and is still buying something at 64. If your service does -one lookup per request, none of this matters and `Shards: 0` is the right answer. +At one lookup per request none of this matters, and `Shards: 0` is the right answer. ## Eviction policies -`LRU` (default) evicts least-recently-used entries until the shard fits. Maintaining that -order means every read takes a write lock. - -`ClearOnFull` drops the whole shard except the entry that just overflowed it. Reads then -need only a read lock, which is worth 20% on a single shard and 40% once the cache is -sharded — a read lock only pays off when the cores taking it are not all queued behind the -same one. This is the right trade when access order is flat (everything in the working set -is used every cycle, so LRU has no information to offer) and refilling is cheap relative to -the bookkeeping. It costs hit rate, though: see the table further down. +`LRU`, the default, evicts least recently used entries, so every read takes a write lock. +`ClearOnFull` drops the whole shard except the entry that overflowed it; reads then take a +read lock, worth 20% on one shard and 40% sharded. It suits flat access orders where +refilling is cheap, and costs hit rate: see the table below. ## The clock -Uncontended, a `Get` that hits costs 47 ns on the machine above, and **27 ns of that is -reading the wall clock** to decide whether the entry has expired. Well over half of a -lookup is `time.Now()`. - -`ClockGranularity` hands that job to a background goroutine and lets lookups read an atomic -instead: +An uncontended hit costs 47 ns, and 27 ns of that is `time.Now()`. `ClockGranularity` has a +background goroutine read the clock instead: ```go sanecache.Options[string, *Article]{ @@ -489,56 +362,42 @@ sanecache.Options[string, *Article]{ } ``` -A hit then costs 20 ns rather than 47, and expiry becomes accurate to within one interval -in either direction — which against a ten-minute TTL is nothing, and against a one-second -TTL is a lot. It is off by default because a TTL that quietly means something other than -what it says is exactly the kind of surprise this library is about. `Close` stops the -goroutine and lookups go back to the wall clock, rather than to a clock that has stopped. +A hit then costs 20 ns, and expiry is accurate to one interval either way: nothing against +ten minutes, a lot against one second, which is why it is off by default. After `Close`, +lookups go back to the wall clock. ## Stats -Counters are built in — no callback interface on the hot path. Poll `Stats()` on your -metrics interval and export it however you like: +Counters are built in. Poll `Stats()`, and `ViewStats()` for every view by name: ```go s := c.Stats() // s.Hits, s.Misses, s.Negatives, s.TypeMisses, s.Evictions, s.Expirations, // s.Replacements, s.Rejections, s.Loads, s.LoadNotFound, s.LoadErrors, s.Coalesced, s.Batches, // s.Refreshes, s.RefreshErrors, s.Entries, s.Bytes, s.HitRate() -``` -`ViewStats()` returns the same kind of snapshot for every view opened on the cache, keyed -by view name, so an exporter needs no list of views of its own: - -```go for name, vs := range c.ViewStats() { // vs.Hits, vs.Misses, vs.Negatives, vs.TypeMisses, vs.Loads, vs.LoadNotFound, // vs.LoadErrors, vs.Coalesced, vs.Refreshes, vs.RefreshErrors, vs.HitRate() } ``` -`Rejections` is the one to alert on: a steady nonzero rate means keys that can never be -cached. `Coalesced` against `Loads` says how much work the single-flight is actually saving -— if they are equal, the cache is cold in a way worth looking at. `LoadErrors` counts loads -that failed; a loader answering `ErrNotFound` did not fail, and is counted in `LoadNotFound` -instead, so an error rate built on `LoadErrors` is not inflated by ids that are simply gone. -`OnEvict` is available separately when entries hold resources that need releasing. +Alert on `Rejections`: keys that can never be cached. `Coalesced` against `Loads` is what +single flight saves. `LoadErrors` leaves out `ErrNotFound`, which is in `LoadNotFound`, so +ids that are gone do not look like failures. ## Lifecycle -A background goroutine sweeps expired entries so they stop occupying budget before anyone -looks them up. `Close()` stops it, along with the clock goroutine if there is one. -Forgetting to call `Close()` does not leak them — dropping the cache stops them too — but -calling it is still better than relying on when the collector gets around to it. - -`DisableCleanup: true` runs without the sweeper; entries then expire only on lookup. +A background goroutine sweeps expired entries off the budget. `Close()` stops it and the +clock goroutine; dropping the cache stops them too. With `DisableCleanup: true` entries +expire only on lookup. ## How it compares -The `benchmarks/` module measures this cache against the ones it would otherwise be -replacing. It is a separate module so the root stays dependency-free; `make bench-compare` -runs it. Same machine as above, Go 1.24, 256-byte values, a budget sized to 4096 of them, -medians of five runs. The sanecache rows are its knobs, each one added to the row above it. +The `benchmarks/` module, separate so the root stays dependency-free, measures this cache +against the ones it would replace: `make bench-compare`. Same machine, Go 1.24, 256-byte +values, a budget of 4096 of them, medians of five runs. Each sanecache row adds a knob to the +row above. | ns/op | serial `Get` | `Get` ×8 | `Set` ×8 | 90/10 ×8 | |---|---:|---:|---:|---:| @@ -553,26 +412,15 @@ medians of five runs. The sanecache rows are its knobs, each one added to the ro | [golang-lru](https://github.com/hashicorp/golang-lru) `v2/expirable` | 54 | 140 | 185 | 168 | | [ttlcache](https://github.com/jellydator/ttlcache) v3 | 63 | 175 | 264 | 193 | -The first column is what one lookup costs; the second is what eight goroutines get out of -the same machine. Read them together, because they say opposite things. Per operation this -cache is *cheaper* than otter, theine and ristretto — they spend their time maintaining a -frequency sketch and a set of ring buffers that only pay off later. What they buy with it is -scaling: otter and theine turn eight cores into 6x to 7x the throughput, this one into 1.6x, -and unsharded into less than 1x, because reads take a lock and locks are where cores queue. - -So the honest shape of it is not "slower". It is: **the same work per operation, and less -of the machine used to do it in parallel.** Whether that matters is a question about the -read rate. At one lookup per request, the gap between 19 ns and 13 ns is six nanoseconds -against a request budget measured in milliseconds. It starts to matter when one request -does thousands of lookups, or when the cache more or less is the service. - -Writes are the other way round: 81 to 90 ns against 121 to 268. An admission policy has to -decide whether to accept each write and update its sketch, and that costs more than taking a -lock does; sturdyc, closest here, has no admission policy either. A write-heavy cache is the -case where this library is simply faster. +Per lookup this cache is *cheaper* than otter, theine and ristretto, which maintain a +frequency sketch and ring buffers. What those buy is scaling: otter and theine turn eight +cores into 6–7x the throughput, this one into 1.6x, because reads take a lock. At one lookup +per request, 19 ns against 13 ns is nothing; it matters when a request does thousands of +lookups. Writes are the other way round, 81–90 ns against 121–268: an admission policy costs +more than a lock. -The number that usually matters more than any of those is how much of a fixed budget each -policy turns into hits. 4096 entries against a key space of 100,000: +What usually matters more is how much of a fixed budget turns into hits. 4096 entries +against 100,000 keys: | %hit | zipf s=1.20 | zipf s=1.01 | zipf + scan | |---|---:|---:|---:| @@ -585,21 +433,14 @@ policy turns into hits. 4096 entries against a key space of 100,000: | golang-lru `expirable` | 87.3 | 65.8 | 78.0 | | ttlcache v3 | 87.3 | 65.8 | 78.0 | -When the hot set fits, every policy looks the same and the extra machinery buys 1.7 points. -When it does not, W-TinyLFU is worth 5.5 points of hit rate over LRU — and 5.5 points of -upstream traffic is worth more than every nanosecond in the table above it. That is the real -reason to pick otter or theine, and it is a better one than throughput. - -The same table prices the cheap read lock: `ClearOnFull` buys its 40% by giving up 7.9 -points of hit rate. It is a trade for caches whose access order is flat, not a free win. -sturdyc makes a milder version of the same trade: when a shard is full it drops the -least recently used tenth of it at once, which costs it 4 points against a plain LRU. +When the hot set fits, policies differ by 1.7 points. When it does not, W-TinyLFU is worth +5.5 points over LRU, and that much upstream traffic outweighs every nanosecond above: the +real reason to pick otter or theine. `ClearOnFull` pays 7.9 points for its cheap reads; +sturdyc, dropping a full shard's oldest tenth at once, pays 4. ### The features are not the difference either -Loading a cold key once, loading many keys in one call, refreshing ahead of expiry and a -budget in something other than entries are not unique to this library. As of otter v2.3.0, -[theine](https://github.com/Yiling-J/theine-go) v0.6.2, +As of otter v2.3.0, [theine](https://github.com/Yiling-J/theine-go) v0.6.2, [sturdyc](https://github.com/viccon/sturdyc) v1.1.6 and ttlcache v3.4.1: | | one load per key | batch load | refresh ahead | budget by cost | "not found" remembered | value over budget | @@ -610,42 +451,34 @@ budget in something other than entries are not unique to this library. As of ott | sturdyc | `GetOrFetch` | `GetOrFetchBatch` | `WithEarlyRefreshes` | — | `WithMissingRecordStorage` | — | | ttlcache v3 | `SuppressedLoader` | — | — | `WithMaxCost` | — | evicts everything, itself last | -They also do things this library does not. otter and theine can save the cache to a file -and load it back; sturdyc retries failed refreshes, coalesces refreshes from many requests -into batches and can sit in front of a distributed store. sturdyc can also hold several -value types in one cache, with typed package-level functions on top, though its capacity is -a count of entries rather than bytes. - -What is left is the combination rather than any single feature: a byte budget that refuses -what does not fit, missing records charged against that budget, and value types with costs -of their own [under one budget](#several-value-types-under-one-budget). +Others do things this one does not: otter and theine save the cache to a file and load it +back; sturdyc retries failed refreshes, batches refreshes across requests, sits in front of a +distributed store, and holds several value types under a count of entries. What is left here +is the combination: a byte budget that refuses what does not fit, missing records charged +against it, and value types with costs of their own +[under one budget](#several-value-types-under-one-budget). ## When to use something else -- You want the best possible hit rate for a given memory budget, and admission policies and - access-frequency estimation are worth the complexity → [otter](https://github.com/maypok86/otter). - The hit-rate table above is the size of the prize, and it is the biggest number on this page. -- Your read path is hot enough that how well a cache scales across cores is a real - difference → otter or [ristretto](https://github.com/dgraph-io/ristretto); with ristretto, - budget for `Wait()` in the tests. -- You call an upstream API and want refreshes retried, refreshes from many requests - coalesced into batches, or a distributed store behind the cache → - [sturdyc](https://github.com/viccon/sturdyc). -- You are caching hundreds of megabytes and GC pressure from millions of live pointers is - your actual problem → [bigcache](https://github.com/allegro/bigcache) or - [freecache](https://github.com/coocood/freecache), which reduce GC scanning by storing entries - in byte buffers with few pointers. These buffers are on the Go heap; reducing pointer - scanning is different from allocating memory outside it. See - [BigCache's storage design](https://github.com/allegro/bigcache#how-it-works) and - [FreeCache's storage design](https://github.com/coocood/freecache#how-it-is-done). -- You just want a bounded LRU with TTL and nothing else → +- The best hit rate for a memory budget is worth admission policies and frequency + estimation → [otter](https://github.com/maypok86/otter). The hit-rate table above is the + size of the prize. +- How a cache scales across cores is a real difference on your read path → otter or + [ristretto](https://github.com/dgraph-io/ristretto); with ristretto, budget for `Wait()` + in the tests. +- You want refreshes retried, refreshes from many requests batched, or a distributed store + behind the cache → [sturdyc](https://github.com/viccon/sturdyc). +- GC pressure from millions of live pointers is your actual problem → + [bigcache](https://github.com/allegro/bigcache) or + [freecache](https://github.com/coocood/freecache), which store entries in byte buffers + with few pointers, still on the Go heap + ([BigCache](https://github.com/allegro/bigcache#how-it-works), + [FreeCache](https://github.com/coocood/freecache#how-it-is-done)). +- A bounded LRU with TTL and nothing else → [hashicorp/golang-lru](https://github.com/hashicorp/golang-lru)'s `v2/expirable`. ## Status v0.7. The API above is what exists and is tested; expect it to move before v1. - -Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for what this -library optimises for before proposing a change. - -MIT licensed. +Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for what this library +optimises for. MIT licensed. diff --git a/cache.go b/cache.go index 0a730fe..04420c4 100644 --- a/cache.go +++ b/cache.go @@ -1,15 +1,8 @@ -// Package sanecache is a small in-memory cache that aims to be predictable -// before it is fast. -// -// Writes are synchronous, so a value is readable the moment Set returns. A value -// that cannot fit is refused with an error instead of being accepted and dropped -// later. Budgets are expressed in bytes, because a limit on the number of entries -// says nothing about the memory a process will use when entries are documents -// rather than integers. "The upstream says this key does not exist" is a first -// class answer rather than a marker smuggled inside the value type. TTLs can -// carry jitter, so a batch of keys warmed by one request does not expire in -// lockstep and stampede the upstream. And a cold key is fetched once rather than -// once per concurrent caller asking for it. +// Package sanecache is a small in-memory cache that is predictable before it is +// fast: writes are synchronous, a value that cannot fit is refused with an error, +// budgets are in bytes, "does not exist" is a first-class answer, TTLs carry +// jitter so that keys warmed together do not expire together, and a cold key is +// loaded once however many callers ask for it. package sanecache import ( @@ -90,135 +83,99 @@ func (r EvictReason) String() string { } } -// Options configures a cache. The zero value is a valid, unbounded, never -// expiring cache; every field below is optional. +// Options configures a cache. Every field is optional: the zero value is an +// unbounded cache whose entries never expire. type Options[K comparable, V any] struct { - // TTL is how long a value stays valid. Zero means entries never expire on - // their own, which only makes sense for a cache bounded by MaxBytes or - // MaxEntries, or one whose entries all get an explicit TTL via SetTTL. + // TTL is how long a value stays valid. Zero means it never expires, which + // suits a bounded cache or one written with SetTTL. TTL time.Duration - // NegativeTTL enables SetNegative and sets how long a cached "does not - // exist" answer lives. It is usually much shorter than TTL: an object that - // does not exist yet may appear at any moment. + // NegativeTTL enables SetNegative and sets how long a "does not exist" answer + // lives. Keep it short: a missing object may appear at any moment. NegativeTTL time.Duration - // Jitter spreads expiry times by up to this percentage in either direction, - // so keys written together do not expire together. Must be 0..100. + // Jitter spreads expiry by up to this percentage either way, so that keys + // written together do not expire together. Must be 0..100. Jitter int - // RefreshAfter starts reloading a value in the background once it is this - // old, so that a key in steady use is replaced before it expires instead of - // going cold and making the next caller wait for the upstream. Zero, the - // default, turns it off. It must be shorter than TTL: a value is still never - // served past its TTL, so the upstream being down shows up as it does - // without refresh, one TTL later. + // RefreshAfter reloads a value in the background once it is this old, so that + // a key in steady use is replaced before it expires instead of making the next + // caller wait. Zero turns it off. It must be shorter than TTL, which still + // holds: a failed refresh is not retried, and the value lives out its TTL. // - // The read that finds a value due returns it at once and starts the load: + // The read that finds a value due returns it and starts its one refresh: // GetOrLoad, GetManyOrLoad and GetOrLoadFunc with their loaders, Get and - // Lookup with Loader or BatchLoader when either is set. A due value gets one - // refresh, whichever read sees it first. If that load fails, the value stays - // until its TTL runs out and the caller after that loads it the usual way; - // a refresh that reports ErrNotFound replaces the value with a negative - // entry when NegativeTTL is set, and leaves it to its TTL otherwise. - // GetManyOrLoad refreshes the keys it finds due in one BatchLoader call. - // - // A refresh is a load like any other: it shares single flight with the key's - // other loads, so a caller that finds the key expired before the refresh - // finishes waits for it rather than starting another, and a Delete, a Clear - // or a successful write keeps its result out. Its context carries the values - // of the read that started it and is never cancelled. Jitter moves the - // refresh with the expiry it belongs to. + // Lookup with Loader or BatchLoader. GetManyOrLoad refreshes the keys it finds + // due in one BatchLoader call. A refresh is an ordinary load to single flight + // and invalidation; its context carries the values of the read that started + // it and is never cancelled. ErrNotFound replaces the value with a negative + // entry when NegativeTTL is set. RefreshAfter time.Duration - // MaxBytes is the total budget in bytes, split evenly across shards. It - // requires Cost. Zero means unbounded. + // MaxBytes is the budget in bytes, split evenly across shards. It requires + // Cost. Zero means unbounded. MaxBytes int64 // MaxEntries caps the number of entries, split evenly across shards. Zero // means unbounded. Prefer MaxBytes unless entries are uniform in size. MaxEntries int - // Cost reports the memory a value occupies, in bytes. It is what MaxBytes is - // measured against, so it should approximate resident size rather than - // serialized size: a decoded struct commonly costs several times its JSON. - // Measure it once rather than guessing (see the README). + // Cost reports the resident size of a value in bytes, which is what MaxBytes + // counts. A decoded struct commonly costs several times its JSON: measure it + // rather than guess (see the README). Cost func(V) int64 - // Loader fetches a value that is not cached. It is what GetOrLoad calls, and - // callers that ask for the same key while it is running share the one call - // instead of each starting their own. - // - // Returning an error that wraps ErrNotFound means the upstream has no such - // key: the answer is cached as a negative entry when NegativeTTL is set, and - // reported to every waiting caller. Any other error is passed through - // unchanged and is not cached. + // Loader fetches a value that is not cached, for GetOrLoad. Callers who ask + // for a key while its load runs share that one call. // - // The context is not any one caller's: it carries the values of the caller - // that started the load, but it is cancelled only once every caller waiting - // for the result has given up. A loader must not call GetOrLoad on the same - // cache and key, which would wait for itself. + // An error wrapping ErrNotFound means the upstream has no such key, and is + // cached as a negative entry when NegativeTTL is set. Other errors are + // returned unchanged and not cached. // - // A load that nobody is waiting for any more is cancelled, but a loader that - // does not watch its context finishes regardless, and its value is cached - // even so. That is what keeps a cache warming when callers time out faster - // than the upstream answers. It cannot put back a stale value: a Delete, a - // Clear or a successful write of the key since the load started, including - // another load's publication, keeps its result out of the cache. + // The context carries the values of the caller that started the load, but is + // cancelled only once every waiting caller has given up. A loader that ignores + // it still caches its result, unless a Delete, a Clear or a successful write of + // the key came after the load started. A loader must not GetOrLoad its own key. Loader func(ctx context.Context, key K) (V, error) - // BatchLoader fetches several keys that are not cached in one call. It is - // what GetManyOrLoad calls with the keys nobody is loading yet, and what - // GetOrLoad calls with a single key when Loader is not set, so that a cache - // with an upstream answering many keys at once needs only this one loader. + // BatchLoader fetches several uncached keys in one call: the keys of + // GetManyOrLoad nobody is loading yet, and the key of GetOrLoad when Loader is + // not set. // - // The result maps each key the upstream has to its value. A requested key - // missing from it means the upstream has no such key, the same answer as a - // Loader returning ErrNotFound; keys that were not requested are ignored. An - // error fails every key of the call, and is neither cached nor combined with - // whatever the map holds. The context follows Loader's rules, counting the - // waiters of every key in the call: it is cancelled once none of them is - // waiting for any key. + // A requested key missing from the result does not exist upstream, as if a + // Loader had returned ErrNotFound; keys not requested are ignored. An error + // fails every key, and nothing from the call is cached. The context is + // cancelled once no caller waits for any of its keys. BatchLoader func(ctx context.Context, keys []K) (map[K]V, error) // Shards splits the cache into independently locked parts, rounded up to a - // power of two. Zero and one both mean a single lock. More shards reduce - // contention but make the budget approximate: each shard gets an equal slice - // of MaxBytes, and an uneven key distribution leaves some of it unused. - // - // The per-shard slice is rounded up, so the shards together are never - // stricter than what was asked for. With small caps that rounding dominates: - // MaxEntries of 2 across 16 shards is one entry per shard, or sixteen in - // total. Keep the cap comfortably larger than the shard count. + // power of two; zero and one both mean a single lock. Each shard gets an equal + // slice of the budget, rounded up, so the budget becomes approximate. Keep it + // well above the shard count: MaxEntries of 2 across 16 shards holds 16. Shards int // Policy selects the eviction strategy. Defaults to LRU. Policy Policy // CleanupInterval is how often a background goroutine drops expired entries. - // Zero picks an interval from the configured TTLs. Expired entries are also - // dropped lazily on lookup, but until they are swept they still count - // against the budget. + // Zero picks it from the TTLs. Lookups drop expired entries too, but until + // then they count against the budget. CleanupInterval time.Duration - // DisableCleanup runs the cache without a background goroutine. Expiry then - // happens only on lookup and on eviction. + // DisableCleanup runs without that goroutine: entries then expire only on + // lookup and on eviction. DisableCleanup bool - // ClockGranularity trades TTL precision for lookup speed. Reading the wall - // clock is about half the cost of a lookup, so a cache under enough load for - // that to show up can have a background goroutine hold the time instead, - // refreshed this often. Expiry is then accurate to within one interval in - // either direction. Zero, the default, reads the clock on every operation. - // - // The goroutine is separate from the sweeper, so this works with - // DisableCleanup. Close stops it, and lookups go back to the wall clock. + // ClockGranularity has a background goroutine read the clock this often, + // instead of every lookup paying for it, which is about half of what a lookup + // costs. Expiry is then accurate to one interval either way. Zero reads the + // clock every time. It works with DisableCleanup; Close returns lookups to the + // wall clock. ClockGranularity time.Duration - // OnEvict, if set, is called for every entry that leaves the cache without - // being explicitly deleted. Negative entries are reported with the zero - // value. It runs outside the shard lock, on the goroutine that caused the - // removal, so it must not block. + // OnEvict is called for every entry that leaves the cache other than by Delete + // or Clear, with the zero value for a negative entry. It runs outside the + // shard lock on the goroutine that caused the removal, so it must not block. OnEvict func(key K, value V, reason EvictReason) // DisableStats skips the counters behind Stats. @@ -231,9 +188,8 @@ type Cache[K comparable, V any] struct { core *core[K, V] } -// core holds everything the background goroutines touch. It is deliberately -// separate from Cache so that a Cache the caller has dropped can be collected -// while those goroutines are still shutting down. +// core holds everything the background goroutines touch, apart from Cache so +// that a dropped Cache can be collected while they shut down. type core[K comparable, V any] struct { shards []*shard[K, V] mask uint64 @@ -255,23 +211,21 @@ type core[K comparable, V any] struct { // every Get. getRefresher refresher - // views holds one set of counters per view name. Views with the same name - // share a namespace in storage, so they share counters too; the map only - // grows with the number of distinct names, which is fixed by the program. + // views holds the counters of each view name: views of one name share keys, + // so they share counters too. viewsMu sync.Mutex views map[string]*counters - // coarse holds the time a background goroutine last read, in unix - // nanoseconds, or nil when the cache reads the clock itself. Zero means the - // goroutine has stopped and the wall clock is authoritative again. + // coarse is the time the clock goroutine last read, in unix nanoseconds; nil + // without ClockGranularity, zero once the goroutine has stopped. coarse *atomic.Int64 stop *stopper } // New builds a cache from o. It panics on options that cannot describe a working -// cache, such as a byte budget without a Cost function: those are programming -// mistakes, and failing at construction beats a cache that silently misbehaves. +// cache, such as a byte budget without Cost: a programming mistake is better +// caught at construction than by a cache that silently misbehaves. func New[K comparable, V any](o Options[K, V]) *Cache[K, V] { switch { case o.Jitter < 0 || o.Jitter > 100: @@ -375,15 +329,13 @@ func (c *Cache[K, V]) Set(key K, value V) error { } // SetTTL caches value under key for ttl, overriding Options.TTL. A ttl of zero -// means the entry never expires on its own. Successful writes supersede outstanding -// loads for the key. +// means the entry never expires on its own. func (c *Cache[K, V]) SetTTL(key K, value V, ttl time.Duration) error { return c.core.setValue(key, value, c.core.valueCost(value), ttl, c.core.refreshAfter, refreshCache) } // SetNegative records that the upstream reports no such key, for the configured -// NegativeTTL. Without it, a template that names a deleted object hits the -// upstream on every single render. A successful write supersedes outstanding loads. +// NegativeTTL. A successful write supersedes outstanding loads. func (c *Cache[K, V]) SetNegative(key K) error { return c.core.setNegative(key, c.core.negativeTTL) } @@ -441,17 +393,14 @@ func (c *Cache[K, V]) Close() { } } -// entryOverhead is what an entry occupies apart from its value and the bytes of a -// string key: the entry itself and its share of the shard map. Measured by -// BenchmarkNegativeEntryMemory at 115-133 bytes for a Cache[string, any], -// depending on how full the map is, and about 16 less for a value type of one -// word; a "does not exist" marker is charged this much plus its key. +// entryOverhead is an entry and its share of the shard map, apart from the value +// and the bytes of a string key: 115-133 bytes for a Cache[string, any] by +// BenchmarkNegativeEntryMemory, depending on how full the map is. const entryOverhead int64 = 128 -// negativeCost is what a "does not exist" marker is charged under a byte budget. -// It holds no value, but it does hold an entry, a map slot and the key; string -// keys are usually the ones worth counting, since a view builds a fresh one for -// every write and the entry keeps it alive. +// negativeCost is what a "does not exist" marker is charged under a byte budget: +// no value, but an entry, a map slot and the key, which for a view is a fresh +// string the entry keeps alive. func negativeCost[K comparable](key K) int64 { if s, ok := any(key).(string); ok { return entryOverhead + int64(len(s)) @@ -490,8 +439,7 @@ func (c *core[K, V]) setNegative(key K, ttl time.Duration, tokens ...*loadToken) return ErrNegativeDisabled } - // A negative entry carries no value, but it is not free either: charging it - // nothing would make it un-evictable under a byte budget. + // Charged nothing, it would never be evicted by a byte budget. var cost int64 if c.cost != nil { cost = negativeCost(key) @@ -677,11 +625,8 @@ func (c *core[K, V]) clockLoop(granularity time.Duration, stop <-chan struct{}) for { select { case <-t.C: - // The clock is read here rather than taken from the tick. A tick - // carries the time it fired, which after a pause or a busy scheduler - // can be several intervals old — and publishing that would make - // expiry lag by the delay on top of the granularity, which is more - // than this option promises. + // Not the tick's own time: after a pause it can be intervals old, + // and expiry would lag by more than the granularity promises. c.coarse.Store(time.Now().UnixNano()) case <-stop: return diff --git a/errors.go b/errors.go index f132880..e08ccfb 100644 --- a/errors.go +++ b/errors.go @@ -4,33 +4,22 @@ import "errors" var ( // ErrTooLarge is returned by Set when the value's cost exceeds the budget of - // the shard it hashes to, so caching it could never succeed. It is reported - // rather than swallowed: a steady trickle of ErrTooLarge means every request - // for those keys goes to the upstream, which is invisible in the hit rate. + // its shard, so that it could never be cached. ErrTooLarge = errors.New("sanecache: value cost exceeds the shard budget") - // ErrNegativeDisabled is returned by SetNegative when Options.NegativeTTL was - // not set. Negative entries without a TTL would pin a "does not exist" answer - // for the lifetime of the process. + // ErrNegativeDisabled is returned by SetNegative without a NegativeTTL: a + // negative entry would otherwise never expire. ErrNegativeDisabled = errors.New("sanecache: negative caching is disabled (Options.NegativeTTL is unset)") // ErrNotFound is how a loader says the upstream has no such key, and how - // GetOrLoad reports that answer back — including when it comes from a cached - // negative entry rather than a fresh call. It is the same error either way, - // so a caller's handling of "no such object" does not depend on whether the - // cache happened to remember it. - // - // A loader may wrap it: GetOrLoad tests with errors.Is and passes the - // loader's own error through to the caller that triggered the load. + // GetOrLoad reports it, from a load or a cached negative entry alike. A + // loader may wrap it: it is tested with errors.Is. ErrNotFound = errors.New("sanecache: the upstream has no such key") - // ErrDisabled is returned by the writes of a view opened on a nil cache. Such a - // view stores nothing, and a write that reported success would be a write the - // next read cannot see. + // ErrDisabled is returned by the writes of a view opened on a nil cache. ErrDisabled = errors.New("sanecache: the view has no cache to write to") - // ErrNoLoader is returned by GetOrLoad and GetManyOrLoad when the cache has - // neither Options.Loader nor Options.BatchLoader, or when the view's - // ViewOptions.Loader was not set, and by GetOrLoadFunc given a nil function. + // ErrNoLoader is returned by GetOrLoad and GetManyOrLoad without a loader, + // and by GetOrLoadFunc given a nil function. ErrNoLoader = errors.New("sanecache: loading requires a loader (Options.Loader, Options.BatchLoader or ViewOptions.Loader)") ) diff --git a/example_test.go b/example_test.go index 5a44066..f6411f2 100644 --- a/example_test.go +++ b/example_test.go @@ -130,9 +130,8 @@ func ExampleCache_GetOrLoad() { Loader: func(_ context.Context, id string) (*article, error) { upstreamCalls.Add(1) if id == "gone" { - // The upstream's own "no such row" is translated once, here, so - // that callers see the same error whether the answer came from - // the upstream or from the negative entry it left behind. + // The upstream's "no such row" is translated once, here, and + // callers see the same error from the negative entry later. return nil, sanecache.ErrNotFound } diff --git a/loader.go b/loader.go index 494c610..d987286 100644 --- a/loader.go +++ b/loader.go @@ -9,20 +9,15 @@ import ( ) // GetOrLoad returns the cached value, calling Options.Loader when there is none, -// or Options.BatchLoader with this one key when there is no Loader. -// Callers that ask for the same key while a load is running wait for it instead -// of starting their own, so a cold key costs one upstream call rather than one -// per concurrent caller. +// or Options.BatchLoader with this one key when there is no Loader. Callers of a +// key whose load is running wait for it instead of starting their own. // -// A cached "does not exist" answer is reported as ErrNotFound without calling -// the loader. A loaded value is stored before this returns, so the next caller -// finds it cached; a value too large for the budget is still returned, counted -// as a rejection rather than quietly retried forever. Invalidation or a successful -// write during loading suppresses publication, but the waiting callers still -// receive the loader result. New callers do not join invalidated loads. -// -// Errors other than ErrNotFound are returned as the loader produced them and are -// not cached, so the next call tries again. +// A cached "does not exist" is reported as ErrNotFound. A loaded value is +// stored before this returns; one too large for the budget is still returned, +// and counted as a rejection. Invalidation or a successful write during the +// load keeps its result out of the cache, though its waiters still get it, and +// new callers start a new load. Other errors are returned as the loader +// produced them and are not cached. func (c *Cache[K, V]) GetOrLoad(ctx context.Context, key K) (V, error) { if c.core.loader == nil { var zero V @@ -36,21 +31,10 @@ func (c *Cache[K, V]) GetOrLoad(ctx context.Context, key K) (V, error) { return c.core.load(ctx, key, c.core.loader) } -// GetOrLoadFunc is GetOrLoad with the loader passed by the caller instead of -// taken from Options. It is for upstream calls that need more than the key to -// make — the parameters a key only summarises, a request the caller has already -// built — which a loader would otherwise have to parse back out of the key. -// -// Everything else is GetOrLoad's: single flight, storage with the cache's Cost -// and TTLs, ErrNotFound kept as a negative entry, the error, cancellation and -// panic policies, and the counters. A load for the key already running, whether -// GetOrLoad, GetManyOrLoad or another GetOrLoadFunc started it, is waited for -// rather than repeated. That is the one rule the method adds: callers of a key -// share whichever load started first, so load must produce what any other -// caller's function would for that key. -// -// Options.Loader and Options.BatchLoader are not needed. A nil load returns -// ErrNoLoader. +// GetOrLoadFunc is GetOrLoad with the loader passed in, for upstream calls that +// need more than the key. A running load of the key, whoever started it, is +// waited for instead, so load must return what any caller's would for that key. +// Options.Loader is not needed; a nil load returns ErrNoLoader. func (c *Cache[K, V]) GetOrLoadFunc(ctx context.Context, key K, load func(context.Context) (V, error)) (V, error) { if load == nil { var zero V @@ -65,9 +49,8 @@ func (c *Cache[K, V]) GetOrLoadFunc(ctx context.Context, key K, load func(contex return c.core.load(ctx, key, loader) } -// cached answers a load from the cache when it can, starting a refresh with -// loader when the value it answers with is due for one. done is false only when -// the key is a miss and the caller still has time to wait for a load. +// cached answers a load from the cache when it can, starting the refresh it +// claims with loader. It reports false only for a miss the caller can wait for. func (c *Cache[K, V]) cached(ctx context.Context, key K, loader func(context.Context, K) (V, error)) (V, bool, error) { v, st, _, refresh := c.core.lookup(key, c.core.refresherWith(loader)) if refresh { @@ -91,8 +74,7 @@ func answered[V any](ctx context.Context, v V, st Status) (V, bool, error) { return v, err != nil, err } -// call is one load in flight. Waiters read val and err only after done is -// closed, which is what publishes them. +// call is one load in flight. Closing done publishes val, err and pan. type call[V any] struct { done chan struct{} val V @@ -100,16 +82,14 @@ type call[V any] struct { pan *loaderPanic token *loadToken - // waiters and cancel are guarded by the group's mutex. The count exists so - // that one caller giving up does not cancel the load the others are waiting - // for; the last one out cancels it. + // Guarded by the group's mutex. The last waiter to give up cancels the load, + // rather than the first. waiters int cancel context.CancelFunc } -// flightGroup tracks the loads in flight for one shard's worth of keys. It has -// its own mutex rather than the shard's: a load is slow, and the map lookup that -// joins one should not queue behind reads of the shard it belongs to. +// flightGroup tracks the loads in flight for one shard's keys, under a mutex of +// its own so that joining a load does not queue behind the shard's reads. type flightGroup[K comparable, V any] struct { mu sync.Mutex calls map[K]*call[V] @@ -131,9 +111,7 @@ func (g *flightGroup[K, V]) running(key K) *call[V] { } // startAndUnlock registers a load of key with one waiter, unlocks g.mu and runs -// the load on a goroutine of its own. The load outlives the caller that starts -// it, so it takes the caller's values but not its cancellation: see -// call.waiters. +// it on its own goroutine, with the caller's values but not its cancellation. func (g *flightGroup[K, V]) startAndUnlock( ctx context.Context, key K, token *loadToken, run func(context.Context, *call[V]), ) *call[V] { @@ -169,13 +147,9 @@ func (c *core[K, V]) load(ctx context.Context, key K, loader func(context.Contex token := s.beginLoad(key) - // The caller's own lookup happened before this lock, and a load that - // finished in between is gone from the map by now — but it publishes its - // value before it leaves, so looking again here, where no load can slip - // past, is what makes "one load per key" exact rather than nearly true. - // Without it, a caller descheduled between the two would start a second load - // for a value that is already cached. Counted without stats, because the - // caller's lookup has already been counted as the miss it was. + // A load that finished since the caller's lookup has left the map but + // published first, so looking again under the lock makes "one load per key" + // exact. Not counted: the caller's lookup already was. if v, st, _, _ := s.get(key, c.now(), refreshNone); st != StatusMiss { s.endLoad(key, token) g.mu.Unlock() @@ -197,10 +171,8 @@ func (c *core[K, V]) load(ctx context.Context, key K, loader func(context.Contex return g.wait(ctx, key, cl) } -// refresh reloads key in the background ahead of its expiry. Nobody waits for -// it, but it is an ordinary flight: a caller that finds the key expired before -// it finishes joins it, and invalidation keeps its result out. A load for the -// key that is already running is left to publish instead. +// refresh reloads key in the background ahead of its expiry, as an ordinary +// flight that callers may join, unless a load of the key is already running. func (c *core[K, V]) refresh(ctx context.Context, key K, loader func(context.Context, K) (V, error)) { idx := c.shardIndex(key) g, s := c.flights[idx], c.shards[idx] @@ -225,9 +197,8 @@ func (c *core[K, V]) countCoalesced(s *shard[K, V]) { } } -// run performs the load and publishes the result. It runs on its own goroutine -// so that a caller can walk away from a load without ending it. refresh says -// the load is a refresh, which is counted apart from loads. +// run performs the load and publishes the result; refresh says it is counted as +// a refresh. func (c *core[K, V]) run( ctx context.Context, g *flightGroup[K, V], s *shard[K, V], key K, cl *call[V], loader func(context.Context, K) (V, error), refresh bool, @@ -254,12 +225,8 @@ func (c *core[K, V]) run( func (g *flightGroup[K, V]) run(key K, cl *call[V], load func() (V, error), record func(loadOutcome)) { defer cl.cancel() - // The loader does not run on a caller's goroutine, so a panic in it would - // take the process down instead of the request that caused it. Carry it to - // the callers, who can recover from it as they would from any other call. - // Counted from the deferred path so that a load that panicked is counted - // too: a service that recovers panics per request would otherwise see a - // loader failing every call and a cache reporting no loads at all. + // A panic here would take down the process rather than the request, so it + // is carried to the callers instead, and counted as a failed load. defer func() { if r := recover(); r != nil { cl.pan = &loaderPanic{value: r, stack: debug.Stack()} @@ -289,9 +256,8 @@ func (g *flightGroup[K, V]) wait(ctx context.Context, key K, cl *call[V]) (V, er } } -// leave drops one waiter. The last one to go cancels the load and takes it out -// of the map, so that the next caller starts a fresh one rather than joining a -// load that is being abandoned. +// leave drops one waiter. The last one cancels the load and takes it out of the +// map, so that the next caller does not join an abandoned load. func (g *flightGroup[K, V]) leave(key K, cl *call[V]) { g.mu.Lock() cl.waiters-- @@ -306,9 +272,8 @@ func (g *flightGroup[K, V]) leave(key K, cl *call[V]) { } } -// finish publishes the result. The call leaves the map first so that a caller -// arriving after this point starts a new load instead of waiting for a result -// that has already been handed out. +// finish publishes the result, taking the call out of the map first so that +// later callers start a new load. func (g *flightGroup[K, V]) finish(key K, cl *call[V]) { g.mu.Lock() g.drop(key, cl) @@ -322,9 +287,8 @@ type loadOutcome uint8 const ( loadOK loadOutcome = iota - // loadNotFound is the upstream saying the key does not exist. It is kept - // apart from failures: a service asking for ids that are gone is working as - // designed, and counting it as an error hides the errors that are real. + // loadNotFound is an answer, not a failure: counted as an error, ids that + // are gone would hide the errors that are real. loadNotFound loadFailed ) @@ -342,8 +306,7 @@ func outcomeOf(err error, pan *loaderPanic) loadOutcome { } } -// loaderPanic carries a panic from the loader's goroutine to the callers waiting -// on it, keeping the stack of where it actually happened. +// loaderPanic carries a loader's panic, with its stack, to the waiting callers. type loaderPanic struct { value any stack []byte diff --git a/loader_batch.go b/loader_batch.go index 409f28b..1accdda 100644 --- a/loader_batch.go +++ b/loader_batch.go @@ -9,43 +9,28 @@ import ( ) // GetManyOrLoad returns the cached values of keys, loading the rest in one -// Options.BatchLoader call. It is GetOrLoad for callers that would otherwise -// fetch keys one at a time from an upstream that answers many at once, such as -// a database query with IN or a pipelined round trip. +// Options.BatchLoader call, for an upstream that answers many keys at once. // -// Each missing key takes part in single flight on its own: a key that some other -// GetOrLoad or GetManyOrLoad is already loading is waited for rather than loaded -// again, and a GetOrLoad arriving while the batch runs waits for the batch. -// Invalidation works per key, exactly as for GetOrLoad: a Delete or a successful -// write of one key keeps the batch's result for that key out of the cache and -// leaves the others alone. +// Single flight and invalidation work per key, as in GetOrLoad: a key already +// loading elsewhere is waited for, and a GetOrLoad of a key in the batch waits +// for the batch. A key that does not exist upstream is absent from the result +// and is not an error. Duplicate keys are looked up once. // -// The result holds every key that has a value. A key the upstream does not have -// is absent from it, whether that answer came from the loader or from a cached -// negative entry; it is not an error. Duplicate keys are looked up once. -// -// The error is the first failure among the keys, in the order they were given, -// and the keys that failed are absent from the result; the others are still -// there. Failures are not cached. If ctx ends first, the result holds what was -// already at hand and the error is ctx.Err(); the loads carry on for any other -// caller waiting for them. A loader panic reaches this caller as it would -// through GetOrLoad. +// The error is the first failure in key order; the keys that failed are absent +// and the others present. If ctx ends first, the result holds what was at hand +// and the error is ctx.Err(), while the loads carry on for other waiters. // // Without a BatchLoader the missing keys are loaded concurrently with -// Options.Loader, one call per key. With neither, it returns ErrNoLoader. -// -// With RefreshAfter, the hits due for a refresh are refreshed together in one -// BatchLoader call of their own, apart from the missing keys: nobody waits for -// them, so they should not hold up the batch somebody does wait for. +// Options.Loader; with neither, it returns ErrNoLoader. Keys due for a refresh +// are refreshed in a BatchLoader call of their own, not holding up this one. func (c *Cache[K, V]) GetManyOrLoad(ctx context.Context, keys []K) (map[K]V, error) { cr := c.core if cr.loader == nil { return nil, ErrNoLoader } - // A warm batch is all hits, and found alone is enough to skip their - // duplicates; the set of keys that were not hits is only built once there is - // one, so that the common case costs one map rather than two. + // notHit is only built once there is a key that was not a hit, so that a + // warm batch costs one map rather than two. found := make(map[K]V, len(keys)) var missing, due []K var notHit map[K]struct{} @@ -98,9 +83,8 @@ func (c *Cache[K, V]) GetManyOrLoad(ctx context.Context, keys []K) (map[K]V, err return found, cr.collect(ctx, waits, found) } -// flight is one key's share in a load: the call its waiters are parked on and -// where that call lives. ctx is the load's context, set only on the flights the -// caller starts itself. +// flight is one key's share in a load. ctx is set only on flights the caller +// starts itself. type flight[K comparable, V any] struct { key K g *flightGroup[K, V] @@ -109,18 +93,15 @@ type flight[K comparable, V any] struct { ctx context.Context } -// batchLoad is the context shared by every key of one batch. It is cancelled -// once every key in it has lost its last waiter: until then somebody still wants -// part of the answer, and the upstream call is the same call whichever part -// that is. +// batchLoad is the context of one batch, cancelled once every key in it has +// lost its last waiter. type batchLoad struct { ctx context.Context cancel context.CancelFunc live atomic.Int64 } -// join adds a key to the batch and returns the cancel function for that key's -// call, which leave invokes when the key's last waiter goes. +// join adds a key to the batch and returns the cancel function of its call. func (b *batchLoad) join() context.CancelFunc { b.live.Add(1) @@ -131,15 +112,10 @@ func (b *batchLoad) join() context.CancelFunc { }) } -// acquire joins or registers a load for each key. Keys whose load published -// between the caller's lookup and here are served from the cache instead, into -// found. It returns every flight to wait for and, among them, the ones this -// caller has to start, along with their batch when there is a batch loader. -// -// A call's cancel is in place before the call is visible to anyone else, as in -// load: a caller that joins and then gives up may be the one to invoke it. It -// cannot bring a fresh call's waiters to zero on its own, though, since this -// caller is one of them until collect. +// acquire joins or registers a load for each key, serving from the cache into +// found the keys whose load published since the caller's lookup. It returns the +// flights to wait for, those of them to start, and their batch, if any. A +// call's cancel is set before anyone else can see the call. func (c *core[K, V]) acquire(ctx context.Context, keys []K, found map[K]V) (waits, fresh []flight[K, V], batch *batchLoad) { for _, key := range keys { idx := c.shardIndex(key) @@ -155,8 +131,7 @@ func (c *core[K, V]) acquire(ctx context.Context, keys []K, found map[K]V) (wait continue } - // The same recheck as load's, for the same reason: without it a caller - // descheduled after its lookup would load a value already cached. + // The same recheck as load's. token := s.beginLoad(key) if v, st, _, _ := s.get(key, c.now(), refreshNone); st != StatusMiss { s.endLoad(key, token) @@ -169,8 +144,7 @@ func (c *core[K, V]) acquire(ctx context.Context, keys []K, found map[K]V) (wait continue } - // The loads outlive this caller, so they do not inherit its - // cancellation: values yes, deadline no. See call.waiters. + // Values, but not cancellation, as in startAndUnlock. cl := &call[V]{done: make(chan struct{}), waiters: 1, token: token} var loadCtx context.Context if c.batchLoader != nil { @@ -233,8 +207,7 @@ func (c *core[K, V]) refreshMany(ctx context.Context, keys []K) { go c.runBatch(batch.ctx, batch.cancel, fresh, true) } -// runBatch performs one batch load and publishes each key's result, the way run -// does for one key. refresh says the batch is a refresh. +// runBatch is run for a batch. func (c *core[K, V]) runBatch(ctx context.Context, cancel context.CancelFunc, fresh []flight[K, V], refresh bool) { defer cancel() defer func() { @@ -243,9 +216,7 @@ func (c *core[K, V]) runBatch(ctx context.Context, cancel context.CancelFunc, fr } }() - // Recovered and counted on the way out for the same reasons as run: the - // batch does not run on any caller's goroutine, and a load that panicked is - // still a load that failed. + // Panics are carried and counted as in flightGroup.run. defer func() { if r := recover(); r != nil { pan := &loaderPanic{value: r, stack: debug.Stack()} @@ -274,9 +245,8 @@ func (c *core[K, V]) runBatch(ctx context.Context, cancel context.CancelFunc, fr v, ok := values[f.key] switch { case err != nil: - // A failed batch fails every key in it, including any the loader - // managed to fill in: part of an answer from a call that reported an - // error is not something to cache. + // Every key fails, even those the map holds: part of an answer + // from a failed call is not worth caching. f.cl.err = err if errors.Is(err, ErrNotFound) { _ = c.setNegative(f.key, c.negativeTTL, f.cl.token) @@ -291,9 +261,8 @@ func (c *core[K, V]) runBatch(ctx context.Context, cancel context.CancelFunc, fr } } -// collect waits for every flight and gathers the results into found. If ctx -// ends first, the flights not yet collected are left, so that a load nobody -// waits for any more can be cancelled. +// collect waits for every flight, gathering results into found. If ctx ends +// first it leaves the rest, so that loads nobody waits for can be cancelled. func (c *core[K, V]) collect(ctx context.Context, waits []flight[K, V], found map[K]V) error { var firstErr error for i, f := range waits { @@ -326,8 +295,7 @@ func (c *core[K, V]) collect(ctx context.Context, waits []flight[K, V], found ma return firstErr } -// loadAsBatch is the loader GetOrLoad uses when only a BatchLoader was given: a -// batch of one, so that a cache needs a single loader for both paths. +// loadAsBatch is GetOrLoad's loader when there is only a BatchLoader. func (c *core[K, V]) loadAsBatch(ctx context.Context, key K) (V, error) { if c.countStats { c.shardFor(key).counters.batches.Add(1) diff --git a/shard.go b/shard.go index 0bc3686..f2ce59a 100644 --- a/shard.go +++ b/shard.go @@ -5,30 +5,28 @@ import ( "sync/atomic" ) -// entry is a single cached item. Entries are linked into an intrusive LRU list -// so that eviction needs no allocation and no side table. +// entry is a cached item, linked into an intrusive LRU list so that eviction +// needs no allocation. type entry[K comparable, V any] struct { key K value V cost int64 expiresAt int64 // unix nanoseconds; 0 means "never expires" - // refreshAt is when a read able to load should start reloading the entry in - // the background, in unix nanoseconds; 0 means never. It is atomic because - // the read that claims the refresh may hold only the read lock, and claiming - // is a swap to 0: one refresh per stored value, however many reads see it due. + // refreshAt is when a read should start reloading the entry, in unix + // nanoseconds; 0 means never. A read claims the refresh by swapping it to 0, + // under what may be only the read lock, hence atomic. refreshAt atomic.Int64 negative bool // upstream said this key does not exist - // byView marks an entry a View wrote. Only a view refreshes it, with its own - // loader: the cache's loader would be handed the view's prefixed key. + // byView marks an entry a View wrote, which only a view may refresh: the + // cache's loader would be handed the prefixed key. byView bool prev, next *entry[K, V] // head of the list is the most recently used entry } -// refresher says which reader may claim an entry's refresh: nobody, the cache -// itself, or a view. It is also what a write records as the entry's owner. +// refresher is who reads an entry, for claiming its refresh, and who wrote it. type refresher uint8 const ( @@ -37,10 +35,9 @@ const ( refreshView ) -// claimRefresh reports whether this read, by a reader that can refresh, is the -// one to refresh e. The caller holds the shard lock, read or write, and checks -// for refreshNone itself: reads that cannot refresh are most reads, and they -// should not pay for a call. +// claimRefresh reports whether a read by by is the one to refresh e. The caller +// holds the shard lock and has ruled out refreshNone, so that most reads skip +// the call. func (e *entry[K, V]) claimRefresh(by refresher, now int64) bool { if e.byView != (by == refreshView) { return false @@ -54,9 +51,7 @@ func (e *entry[K, V]) expired(now int64) bool { return e.expiresAt != 0 && now >= e.expiresAt } -// shard is an independently locked slice of the cache. A cache with one shard -// is a plain map behind a single lock; more shards trade exact budget accounting -// for less lock contention. +// shard is an independently locked slice of the cache. type shard[K comparable, V any] struct { mu sync.RWMutex items map[K]*entry[K, V] @@ -71,23 +66,16 @@ type shard[K comparable, V any] struct { maxEntries int policy Policy - // Counters live on the shard rather than on the cache because one counter is - // one cache line, and every core in the process writes to it on every - // lookup: that single line held reads to a fifth of their throughput however - // many shards they were spread across. The lock was never the limit. - // - // The padding matters as much as the split does. Sharing a line with the - // fields above, which every read reads, a counter write invalidates that - // line for every other core — and a read lock then costs more than the - // exclusive one it was there to avoid. Measured on a single shard under - // ClearOnFull: 161 ns padded down to 109. + // Counters per shard, padded off the fields every read reads: one shared + // counter line held reads to a fifth of their throughput however many shards + // they used, and without padding a read lock cost more than the write lock it + // avoids (padding took one ClearOnFull shard from 161 ns to 109). _ [cacheLine]byte counters counters _ [cacheLine]byte } -// cacheLine is 64 bytes on every architecture this is likely to run on, and -// being wrong about it costs padding rather than correctness. +// cacheLine is 64 bytes almost everywhere; being wrong costs only padding. const cacheLine = 64 func newShard[K comparable, V any](maxBytes int64, maxEntries int, policy Policy) *shard[K, V] { @@ -99,16 +87,13 @@ func newShard[K comparable, V any](maxBytes int64, maxEntries int, policy Policy } } -// get returns the entry's value along with its status. expired reports that the -// lookup landed on an entry that had already expired, so the caller can account -// for it separately from a plain miss. refresh reports that the hit was due for a -// refresh and that this reader, being the kind given by by, claimed it. +// get returns the entry's value and status. expired reports a miss on an entry +// that had expired; refresh, that the hit was due and the read by by claimed it. func (s *shard[K, V]) get(key K, now int64, by refresher) (_ V, _ Status, expired, refresh bool) { var zero V - // Under ClearOnFull nothing is reordered on access, so reads take only the - // read lock. That is the whole point of the policy: keeping LRU order costs - // a write lock on every read. + // ClearOnFull reorders nothing on access, so reads take only the read lock: + // that is the point of the policy. if s.policy == ClearOnFull { s.mu.RLock() e, ok := s.items[key] @@ -119,10 +104,8 @@ func (s *shard[K, V]) get(key K, now int64, by refresher) (_ V, _ Status, expire if e.expired(now) { s.mu.RUnlock() - // Drop it under the write lock rather than leaving it for the - // sweeper: an expired entry that stays in the map would be counted - // as an expiration again on every lookup, and would go on holding - // budget in the meantime. Only the goroutine that removes it counts it. + // Dropped now, or every lookup until the sweep would count it as an + // expiration again. Only the goroutine that removes it counts it. s.mu.Lock() cur, still := s.items[key] removed := still && cur == e @@ -170,9 +153,8 @@ func (s *shard[K, V]) get(key K, now int64, by refresher) (_ V, _ Status, expire return e.value, StatusHit, false, refresh } -// set stores e, returning the entry it replaced (if any) and the entries evicted -// to stay inside the budget. A value that cannot fit on its own is rejected with -// ErrTooLarge rather than stored and silently dropped later. +// set stores e, returning the entry it replaced and those evicted to make room. +// A value that cannot fit on its own is refused with ErrTooLarge. func (s *shard[K, V]) set(e *entry[K, V], tokens ...*loadToken) (replaced *entry[K, V], victims []*entry[K, V], err error) { if s.maxBytes > 0 && e.cost > s.maxBytes { return nil, nil, ErrTooLarge @@ -246,9 +228,8 @@ func (s *shard[K, V]) stats() (entries int, bytes int64) { return len(s.items), s.bytes } -// evict brings the shard back inside its budget. keep is the entry that was just -// stored: ClearOnFull wipes everything else rather than picking victims, but -// dropping the write that triggered the wipe would only force an immediate refetch. +// evict brings the shard back inside its budget, never evicting keep, the entry +// just stored: ClearOnFull wipes everything else. func (s *shard[K, V]) evict(keep *entry[K, V]) []*entry[K, V] { if !s.over() { return nil @@ -324,8 +305,8 @@ func (s *shard[K, V]) unlink(e *entry[K, V]) { e.prev, e.next = nil, nil } -// loadToken connects a flight to mutations of its storage key, including writes -// through other views. Validity is checked under the storage lock when publishing. +// loadToken lets a write or delete of a storage key, through any view, stop the +// key's loads from publishing; it is checked under the shard lock. type loadToken struct{ valid atomic.Bool } func (s *shard[K, V]) beginLoad(key K) *loadToken { diff --git a/stats.go b/stats.go index a013296..b40f818 100644 --- a/stats.go +++ b/stats.go @@ -2,8 +2,7 @@ package sanecache import "sync/atomic" -// Stats is a snapshot of the cache counters. Counters are cumulative since the -// cache was created; Entries and Bytes are instantaneous. +// Stats is a snapshot of the cache counters, cumulative except Entries and Bytes. type Stats struct { Hits int64 // lookups that returned a value Misses int64 // lookups that found nothing @@ -13,41 +12,28 @@ type Stats struct { Replacements int64 // entries overwritten by a later Set Rejections int64 // Set calls refused with ErrTooLarge - // TypeMisses counts view lookups that found an entry holding some other - // type. Hits counts those too, because the cache did have the key; the pair - // is what tells a namespace collision apart from a plain miss. + // TypeMisses counts view lookups that found another type. Hits counts them + // too, since the cache had the key. TypeMisses int64 Loads int64 // keys loaded by Loader or BatchLoader, successfully or not - // LoadNotFound counts the loads that ended in ErrNotFound, a key missing - // from a batch answer included. It is an answer rather than a failure, so - // LoadErrors does not count it. + // LoadNotFound counts loads that ended in ErrNotFound, keys missing from a + // batch answer included; LoadErrors does not. LoadNotFound int64 LoadErrors int64 // loads that failed: any other error, or a panic - // Batches counts BatchLoader calls. Against Loads it says how many keys an - // upstream call carries on average. - Batches int64 - // Coalesced counts the GetOrLoad calls that another caller's load spared - // from starting one of their own, whether they waited for it or arrived just - // after it published. Against Loads it says how much the single flight is - // actually saving. - Coalesced int64 - // Refreshes counts the reloads RefreshAfter started that finished, however - // they ended, a key at a time as Loads does. They are not in Loads: Loads is - // what misses cost the upstream, and Refreshes is what keeping hot keys warm - // costs it instead. + Batches int64 // BatchLoader calls; Loads/Batches is keys per call + Coalesced int64 // GetOrLoad calls spared a load by another caller's + // Refreshes counts finished refreshes, by key. Loads does not include them: + // it is what misses cost. Refreshes int64 - // RefreshErrors counts the refreshes that failed or panicked. The value they - // meant to replace stays until its TTL runs out. A refresh that ended in - // ErrNotFound is not one of them. + // RefreshErrors counts refreshes that failed or panicked, not ErrNotFound. RefreshErrors int64 Entries int // entries currently held, expired-but-not-yet-swept included Bytes int64 // sum of the costs of those entries } -// HitRate reports hits as a fraction of all lookups. A cached negative answer -// counts as a hit: it saved the same upstream call a positive one would have. +// HitRate reports hits, cached negatives included, as a fraction of lookups. func (s Stats) HitRate() float64 { total := s.Hits + s.Misses + s.Negatives if total == 0 { diff --git a/view.go b/view.go index 0a0a83d..23c0de1 100644 --- a/view.go +++ b/view.go @@ -9,69 +9,45 @@ import ( ) // viewSeparator joins a view's name to the caller's key. Names may not contain -// it, which is what makes the join unambiguous: "article:1:2" can only be key -// "1:2" in view "article". +// it, so "article:1:2" can only be key "1:2" in view "article". const viewSeparator = ":" // ViewOptions configures a view. Only Name is required. type ViewOptions[T any] struct { - // Name identifies the view and namespaces its keys: the view stores under - // Name + ":" + key. Two views therefore cannot collide, and an OnEvict - // handler on the cache can tell whose entry it is looking at. It must not be - // empty and must not contain ":". + // Name namespaces the view's keys: it stores under Name + ":" + key. It must + // not be empty or contain ":". Name string - // Loader fetches an uncached value using the caller's key without the name - // prefix. It follows Options.Loader's error, cancellation and panic policy. - // Concurrent loads coalesce within this View instance; reuse the instance - // to share in-flight work. Separate views (even with the same name) and the - // underlying cache have independent loaders and in-flight work. - // A loader must not recursively load the same key through this view. + // Loader fetches an uncached value by the caller's key, without the prefix, + // under Options.Loader's rules. Loads coalesce within one View instance, not + // across views of the same name or with the cache's own loads. A loader must + // not load its own key through this view. Loader func(context.Context, string) (T, error) - // Cost reports the memory a value of this view occupies, in bytes, the way - // Options.Cost does for the cache as a whole. A view that sets it is spared - // the type switch that a shared Cost func(any) int64 turns into once a - // budget holds several types. Unset, the cache's own Cost is used. + // Cost is Options.Cost for this view's values, sparing the cache's Cost a type + // switch. Unset, the cache's Cost is used. Cost func(T) int64 - // TTL is how long this view's values stay valid. Zero takes the cache's TTL; - // SetTTL still overrides both. - // - // A view with a much shorter TTL than the cache is worth a word of warning: - // the sweeper's interval is chosen from the cache's TTLs, so those entries - // may sit on the budget after expiring until a lookup or a sweep finds them. - // Set Options.CleanupInterval when that matters. + // TTL is how long this view's values stay valid; zero takes the cache's. The + // sweep interval follows the cache's TTLs, so with a much shorter TTL here, + // set Options.CleanupInterval. TTL time.Duration - // NegativeTTL is how long this view's "does not exist" answers live. Zero - // takes the cache's NegativeTTL, and if that is unset too, SetNegative on - // this view reports ErrNegativeDisabled. + // NegativeTTL is how long this view's "does not exist" answers live; zero + // takes the cache's. NegativeTTL time.Duration - // RefreshAfter is Options.RefreshAfter for this view's values, refreshed with - // this view's loaders: Loader for GetOrLoad, Get and Lookup, the caller's - // function for GetOrLoadFunc. It must be shorter than the view's TTL. Unlike - // TTL it is not taken from the cache when zero, since the cache's would have - // to fit every view's TTL; zero means this view does not refresh, and the - // cache's RefreshAfter never touches a view's values. + // RefreshAfter is Options.RefreshAfter for this view, with its own loaders. + // It must be shorter than the view's TTL. Zero means no refresh: the cache's + // RefreshAfter is not inherited and never applies to a view's values. RefreshAfter time.Duration } -// View is a typed window onto a cache that holds values of several types under -// one byte budget. The cache is declared as Cache[string, any]; each view fixes -// one value type, namespaces its keys, and counts its own hits. -// -// This is the shape that a budget in bytes forces. One cache per type would mean -// one budget per type, and splitting a fixed amount of memory between types up -// front is exactly the guess the byte budget was meant to avoid: the split that -// was right at deploy time is wrong by the next traffic pattern. A view is a -// function rather than a method on Cache because a method cannot introduce a -// type parameter of its own. -// -// A view costs one string join per operation on top of the cache underneath. -// While the name and key together fit in 32 bytes the compiler keeps that on the -// stack; past it, every read allocates. +// View is a typed window onto a Cache[string, any] that holds several value types +// under one byte budget: one cache per type would mean splitting the memory +// between types up front. Each view fixes a type, namespaces its keys and counts +// its own hits. It costs a string join per operation, which allocates once name +// and key exceed 32 bytes. type View[T any] struct { cache *Cache[string, any] name string @@ -82,8 +58,7 @@ type View[T any] struct { refreshAfter time.Duration countStats bool - // getRefresher is who Get and Lookup read as: refreshView with RefreshAfter - // and a Loader, refreshNone otherwise. Worked out once, as the cache does. + // getRefresher is refresherWith(loader), worked out once. getRefresher refresher loader func(context.Context, string) (T, error) @@ -93,15 +68,13 @@ type View[T any] struct { stats *counters } -// NewView opens a view named o.Name onto c. It panics on a name that cannot keep -// views apart, for the same reason New panics on a budget it cannot honour. +// NewView opens a view named o.Name onto c. It panics on invalid options, as New +// does. // -// A nil c opens a view with caching switched off, for the configuration where a -// TTL of zero means "do not cache": code keeps calling GetOrLoad either way. Such -// a view runs the loader on every call, still sharing one call among callers who -// arrive while it runs, and remembers nothing, "does not exist" included. Its -// lookups are misses, Delete reports false, and writes fail with ErrDisabled. -// Its counters are its own. +// A nil c switches caching off, so that code can keep calling GetOrLoad: the +// loader runs on every call, still shared by concurrent callers, and nothing is +// remembered. Lookups miss, Delete reports false, and writes fail with +// ErrDisabled. func NewView[T any](c *Cache[string, any], o ViewOptions[T]) *View[T] { switch { case o.Name == "": @@ -160,8 +133,7 @@ func NewView[T any](c *Cache[string, any], o ViewOptions[T]) *View[T] { return v } -// Name returns the view's name, which is also the prefix its keys carry in the -// underlying cache. +// Name returns the view's name, the prefix of its keys in the cache. func (v *View[T]) Name() string { return v.name } // Get returns the cached value. As with Cache.Get, a cached "does not exist" @@ -172,18 +144,15 @@ func (v *View[T]) Get(key string) (T, bool) { return val, st == StatusHit } -// Lookup returns the cached value and how the view answered. An entry holding -// some other type is reported as a miss and counted as a TypeMiss: the value is -// unusable here, so the caller has to go to the upstream either way. With -// RefreshAfter and a Loader, a value due for a refresh starts one. +// Lookup returns the cached value and how the view answered. An entry of another +// type is a miss, counted as a TypeMiss. With RefreshAfter and a Loader, a value +// due for a refresh starts one. func (v *View[T]) Lookup(key string) (T, Status) { - // background rather than context.Background(): the call is what would keep - // this from inlining into Get, and that costs every view read a call. + // Not context.Background(): that call would keep this from inlining. return v.lookup(background, key, v.getRefresher, v.loader) } -// background is the context of the refreshes Get and Lookup start, which have no -// caller's context to carry. +// background is the context of refreshes that Get and Lookup start. var background = context.Background() // lookup is Lookup as reader by, refreshing with loader. @@ -232,8 +201,7 @@ func (v *View[T]) Set(key string, value T) error { return v.SetTTL(key, value, v.ttl) } -// SetTTL caches value under key for ttl, overriding both the view's and the -// cache's TTL. A ttl of zero means the entry never expires on its own. +// SetTTL caches value under key for ttl; zero means it never expires. func (v *View[T]) SetTTL(key string, value T, ttl time.Duration) error { if v.cache == nil { return ErrDisabled @@ -257,9 +225,8 @@ func (v *View[T]) SetNegativeTTL(key string, ttl time.Duration) error { return v.cache.core.setNegative(v.prefix+key, ttl) } -// Delete removes key from this view and reports whether it was present. It also -// invalidates outstanding loads for the namespaced key, including those in other -// views or the parent cache. Existing waiters still receive their loader result. +// Delete is Cache.Delete for this view's key, invalidating its loads in any view +// or the cache. func (v *View[T]) Delete(key string) bool { if v.cache == nil { return false @@ -268,16 +235,14 @@ func (v *View[T]) Delete(key string) bool { return v.cache.Delete(v.prefix + key) } -// Stats returns a snapshot of the counters for this view's name. Views opened -// with the same name on the same cache share them, as they share the keys. The -// cache's own Stats counts the same lookups across every view. +// Stats returns the counters of this view's name, shared by every view of that +// name on the cache. The cache's Stats counts the same lookups too. func (v *View[T]) Stats() ViewStats { return v.stats.viewStats() } -// ViewStats returns a snapshot of the counters of every view opened on the -// cache, by view name. It is what a metrics exporter polls: the cache knows its -// views, so the application does not need a registry of its own. +// ViewStats returns the counters of every view opened on the cache, by name, so +// that an exporter needs no registry of its own. func (c *Cache[K, V]) ViewStats() map[string]ViewStats { c.core.viewsMu.Lock() defer c.core.viewsMu.Unlock() @@ -290,8 +255,7 @@ func (c *Cache[K, V]) ViewStats() map[string]ViewStats { return stats } -// viewCounters returns the counters for a view name, creating them on first use. -// A view uses only some of them. +// viewCounters returns the counters of a view name, creating them on first use. func (c *core[K, V]) viewCounters(name string) *counters { c.viewsMu.Lock() defer c.viewsMu.Unlock() @@ -331,17 +295,14 @@ func (v *View[T]) count(c *atomic.Int64) { } } -// ViewStats is a snapshot of the counters for one view name, cumulative since -// the first view with that name was opened on the cache. +// ViewStats is a snapshot of the counters of one view name. type ViewStats struct { Hits int64 // lookups that returned a value of this view's type Misses int64 // lookups that found nothing Negatives int64 // lookups that found a cached "does not exist" - // TypeMisses counts lookups that found an entry holding another type. It is - // a bug detector rather than a routine metric: with keys namespaced by view - // name, the only ways to get one are two views sharing a name and writes - // made straight to the underlying cache. + // TypeMisses counts lookups that found another type: a bug detector, since + // only two views of one name or writes straight to the cache cause them. TypeMisses int64 Loads int64 // completed loader calls, including errors and panics diff --git a/view_loader.go b/view_loader.go index a49c6ff..baa930d 100644 --- a/view_loader.go +++ b/view_loader.go @@ -5,13 +5,9 @@ import ( "errors" ) -// GetOrLoad returns a typed cached value or calls ViewOptions.Loader. It follows -// Cache.GetOrLoad's error and cancellation policy, storing results with this -// view's Cost, TTL and NegativeTTL under the shared cache's budget. A wrong-type -// entry is a miss and can be replaced by the loaded value. -// -// Without a view loader it returns ErrNoLoader, even for a cached key; the -// underlying cache's loader is never used. Loads coalesce per View instance. +// GetOrLoad is Cache.GetOrLoad with ViewOptions.Loader, storing with this view's +// Cost and TTLs; an entry of another type is a miss the load replaces. Without +// a view loader it returns ErrNoLoader, even for a cached key. func (v *View[T]) GetOrLoad(ctx context.Context, key string) (T, error) { if v.loader == nil { var zero T @@ -26,15 +22,8 @@ func (v *View[T]) GetOrLoad(ctx context.Context, key string) (T, error) { return v.loadWith(ctx, key, v.loader) } -// GetOrLoadFunc is GetOrLoad with the loader passed by the caller instead of -// taken from ViewOptions, as Cache.GetOrLoadFunc is for the cache. Results are -// stored with this view's Cost and TTLs. Loads are shared with this View -// instance's GetOrLoad, so load must produce what the view's loader, or any -// other caller's function, would for that key. -// -// ViewOptions.Loader is not needed. A nil load returns ErrNoLoader. On a view -// with caching switched off, load runs on every call that does not find one -// already running. +// GetOrLoadFunc is Cache.GetOrLoadFunc for this view, sharing loads with its +// GetOrLoad. A nil load returns ErrNoLoader. func (v *View[T]) GetOrLoadFunc(ctx context.Context, key string, load func(context.Context) (T, error)) (T, error) { if load == nil { var zero T @@ -64,8 +53,7 @@ func (v *View[T]) loadWith(ctx context.Context, key string, loader func(context. return v.load(ctx, key, loader) } -// loadUncached is load for a view without a cache: callers still share one call -// while it runs, but its result, "does not exist" included, is not kept. +// loadUncached is load for a view without a cache: shared, but not kept. func (v *View[T]) loadUncached(ctx context.Context, key string, loader func(context.Context, string) (T, error)) (T, error) { g := v.flights[0] @@ -103,8 +91,7 @@ func (v *View[T]) load(ctx context.Context, key string, loader func(context.Cont return g.wait(ctx, key, cl) } token := s.beginLoad(fullKey) - // Recheck without counting another lookup: a load may have published since - // the caller's miss. A value of a different type still needs a typed load. + // load's recheck; a value of another type still needs loading. raw, st, _, _ := s.get(fullKey, c.now(), refreshNone) val, typed := raw.(T) if st == StatusNegative || (st == StatusHit && typed) { @@ -144,8 +131,7 @@ func (v *View[T]) refresh(ctx context.Context, key string, loader func(context.C }) } -// run is Cache's run for a view's key: the result is stored under the prefixed -// key, with the view's cost and lifetimes, and counted for the view too. +// run is Cache's run for a view's key, counted for the view too. func (v *View[T]) run( ctx context.Context, g *flightGroup[string, T], s *shard[string, any], key string, cl *call[T], loader func(context.Context, string) (T, error), refresh bool, From 36173a478824d22508f93f0bd699a3fb4f5e0dcc Mon Sep 17 00:00:00 2001 From: Andrey Ryabokon Date: Fri, 9 Oct 2026 08:18:19 +0300 Subject: [PATCH 5/9] Share test helpers for the manual clock and sorted batches --- cache_test.go | 11 +++++++++ refresh_test.go | 56 ++++++++++++++------------------------------- view_loader_test.go | 10 ++++---- 3 files changed, 32 insertions(+), 45 deletions(-) diff --git a/cache_test.go b/cache_test.go index 69201a3..fc35c7c 100644 --- a/cache_test.go +++ b/cache_test.go @@ -4,6 +4,7 @@ import ( "fmt" "strings" "sync" + "sync/atomic" "testing" "time" ) @@ -25,6 +26,16 @@ func waitFor(t *testing.T, what string, cond func() bool) { } } +// manualClock makes the cache read its time from a counter the test moves, so +// that expiry is a step in the test rather than a sleep. Install it before the +// first write. +func manualClock[K comparable, V any](c *Cache[K, V]) (advance func(time.Duration)) { + c.core.coarse = new(atomic.Int64) + c.core.coarse.Store(time.Now().UnixNano()) + + return func(d time.Duration) { c.core.coarse.Add(int64(d)) } +} + func TestGetSet(t *testing.T) { c := New(Options[string, int]{}) defer c.Close() diff --git a/refresh_test.go b/refresh_test.go index e80fa8f..e1a252a 100644 --- a/refresh_test.go +++ b/refresh_test.go @@ -17,16 +17,6 @@ const ( refreshAfter = 5 * time.Minute ) -// manualClock makes the cache read its time from a counter the test moves, so -// that "the value is now due" is a step in the test rather than a sleep. It must -// be installed before the first write. -func manualClock[K comparable, V any](c *Cache[K, V]) (advance func(time.Duration)) { - c.core.coarse = new(atomic.Int64) - c.core.coarse.Store(time.Now().UnixNano()) - - return func(d time.Duration) { c.core.coarse.Add(int64(d)) } -} - // flying reports whether a load for key is registered, which refresh does // before it returns: a test can tell "no refresh was started" without waiting. func flying[V any](c *Cache[string, V], key string) bool { @@ -379,17 +369,13 @@ func TestGetManyOrLoadRefreshesDueKeysInOneBatch(t *testing.T) { waitFor(t, "the refresh batch", func() bool { return c.Stats().Refreshes == 3 }) rec.mu.Lock() - batches := slices.Clone(rec.batches) - rec.mu.Unlock() - sorted := make([]string, 0, len(batches)) - for _, b := range batches { - b = slices.Clone(b) - slices.Sort(b) - sorted = append(sorted, strings.Join(b, ",")) + var batches []string + for _, b := range rec.batches { + batches = append(batches, strings.Join(sorted(b), ",")) } - slices.Sort(sorted) - if want := []string{"a,b,c", "a,b,c", "d"}; !slices.Equal(sorted, want) { - t.Fatalf("batches %v, want %v", sorted, want) + rec.mu.Unlock() + if want := []string{"a,b,c", "a,b,c", "d"}; !slices.Equal(sorted(batches), want) { + t.Fatalf("batches %v, want %v", batches, want) } if st := c.Stats(); st.Batches != 3 || st.Loads != 4 { t.Fatalf("stats: %+v", st) @@ -529,23 +515,20 @@ func TestRefreshLeavesARunningLoadToPublish(t *testing.T) { return 0, nil } - running := func(c *core[string, int], key string) *call[int] { - token := &loadToken{} - token.valid.Store(true) - cl := &call[int]{done: make(chan struct{}), token: token} - g := c.flights[c.shardIndex(key)] - g.mu.Lock() + running := func(g *flightGroup[string, int], key string) *call[int] { + cl := &call[int]{done: make(chan struct{}), token: &loadToken{}} + cl.token.valid.Store(true) g.calls[key] = cl - g.mu.Unlock() return cl } c := New(Options[string, int]{TTL: refreshTTL, RefreshAfter: refreshAfter, Loader: never, DisableCleanup: true}) defer c.Close() - cl := running(c.core, "k") + g := c.core.flights[c.core.shardIndex("k")] + cl := running(g, "k") c.core.refresh(context.Background(), "k", never) - if g := c.core.flights[c.core.shardIndex("k")]; g.calls["k"] != cl { + if g.calls["k"] != cl { t.Fatal("refresh replaced the running load") } @@ -558,22 +541,17 @@ func TestRefreshLeavesARunningLoadToPublish(t *testing.T) { }, }) defer b.Close() - running(b.core, "x") - running(b.core, "y") + running(b.core.flights[b.core.shardIndex("x")], "x") + running(b.core.flights[b.core.shardIndex("y")], "y") b.core.refreshMany(context.Background(), []string{"x", "y"}) vc := New(Options[string, any]{TTL: refreshTTL, DisableCleanup: true}) defer vc.Close() v := NewView(vc, ViewOptions[int]{Name: "v", RefreshAfter: refreshAfter, Loader: never}) - g := v.flights[vc.core.shardIndex("v:k")] - token := &loadToken{} - token.valid.Store(true) - vcl := &call[int]{done: make(chan struct{}), token: token} - g.mu.Lock() - g.calls["k"] = vcl - g.mu.Unlock() + vg := v.flights[vc.core.shardIndex("v:k")] + vcl := running(vg, "k") v.refresh(context.Background(), "k", never) - if g.calls["k"] != vcl { + if vg.calls["k"] != vcl { t.Fatal("a view refresh replaced the running load") } diff --git a/view_loader_test.go b/view_loader_test.go index bbed03f..8256821 100644 --- a/view_loader_test.go +++ b/view_loader_test.go @@ -53,8 +53,7 @@ func TestViewLoaderPolicies(t *testing.T) { c := New(Options[string, any]{TTL: time.Hour, NegativeTTL: time.Hour, MaxBytes: 1000, Cost: func(any) int64 { return 1 }, DisableStats: disabled, DisableCleanup: true}) defer c.Close() // Drive expiry explicitly so scheduler delays cannot expire recheck fixtures. - c.core.coarse = new(atomic.Int64) - c.core.coarse.Store(time.Now().UnixNano()) + advance := manualClock(c) var calls int v := NewView(c, ViewOptions[string]{Name: "text", TTL: 100 * time.Millisecond, NegativeTTL: 100 * time.Millisecond, Cost: func(s string) int64 { return int64(len(s)) }, Loader: func(_ context.Context, key string) (string, error) { calls++ @@ -90,7 +89,7 @@ func TestViewLoaderPolicies(t *testing.T) { if _, err := v.GetOrLoad(ctx, "gone"); err != ErrNotFound { t.Fatalf("cached negative: %v", err) } - c.core.coarse.Add(int64(150 * time.Millisecond)) + advance(150 * time.Millisecond) if _, st := v.Lookup("k"); st != StatusMiss { t.Fatal("view TTL ignored") } @@ -284,8 +283,7 @@ func TestViewLoadsShareBudgetAndInheritOptions(t *testing.T) { t.Run(fmt.Sprint(policy), func(t *testing.T) { c := New(Options[string, any]{MaxBytes: 512, Cost: func(any) int64 { return 256 }, TTL: time.Minute, NegativeTTL: time.Minute, DisableCleanup: true, Policy: policy}) defer c.Close() - c.core.coarse = new(atomic.Int64) - c.core.coarse.Store(time.Now().UnixNano()) + advance := manualClock(c) ints := NewView(c, ViewOptions[int]{Name: "int", Loader: func(context.Context, string) (int, error) { return 7, nil }}) texts := NewView(c, ViewOptions[string]{Name: "text", Loader: func(_ context.Context, key string) (string, error) { if key == "gone" { @@ -315,7 +313,7 @@ func TestViewLoadsShareBudgetAndInheritOptions(t *testing.T) { if _, st := texts.Lookup("gone"); st != StatusNegative { t.Fatal("negative TTL not inherited") } - c.core.coarse.Add(int64(2 * time.Minute)) + advance(2 * time.Minute) if _, st := texts.Lookup("next"); st != StatusMiss { t.Fatal("TTL not inherited") } From 1eb548b93ce45b8f58212ba15651d4d0d2e21d99 Mon Sep 17 00:00:00 2001 From: Andrey Ryabokon Date: Fri, 9 Oct 2026 08:31:44 +0300 Subject: [PATCH 6/9] Let a value's cost come with it instead of from Cost --- CHANGELOG.md | 3 ++ README.md | 3 ++ cache.go | 15 +++++--- cost_test.go | 91 +++++++++++++++++++++++++++++++++++++++++++++++++ loader.go | 66 ++++++++++++++++++++++++++++------- refresh_test.go | 4 +-- view.go | 21 ++++++++---- view_loader.go | 66 +++++++++++++++++++++++------------ 8 files changed, 223 insertions(+), 46 deletions(-) create mode 100644 cost_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 00d2f04..e56f2e4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,9 @@ While the major version is 0, the public API may change in any release. leaves the value to its TTL, which is never extended. Off by default; it must be shorter than the TTL. - `Stats.Refreshes` and `Stats.RefreshErrors`, also in `ViewStats`, apart from `Loads`. +- `SetWithCost` and `GetOrLoadFuncWithCost` on `Cache` and `View`: the value's cost comes + with it instead of from the `Cost` function, for values whose size is known only where + they are produced, such as the length of the response they were decoded from. ### Changed diff --git a/README.md b/README.md index 7cfba60..fd2c2ab 100644 --- a/README.md +++ b/README.md @@ -171,6 +171,9 @@ page, err := c.GetOrLoadFunc(ctx, pageKey(path, params), func(ctx context.Contex It shares loads with `GetOrLoad`: callers of a key wait for whichever load started first, so the function must return what any caller's would for that key. Views have it too. +`GetOrLoadFuncWithCost` and `SetWithCost` take the value's cost along with it, for when +the size is known only where the value is produced, such as the response it was decoded +from. ## Invalidation while loading diff --git a/cache.go b/cache.go index 04420c4..b848e04 100644 --- a/cache.go +++ b/cache.go @@ -203,7 +203,7 @@ type core[K comparable, V any] struct { onEvict func(K, V, EvictReason) countStats bool - loader func(context.Context, K) (V, error) + loader loadFunc[K, V] batchLoader func(context.Context, []K) (map[K]V, error) flights []*flightGroup[K, V] @@ -258,11 +258,11 @@ func New[K comparable, V any](o Options[K, V]) *Cache[K, V] { cost: o.Cost, onEvict: o.OnEvict, countStats: !o.DisableStats, - loader: o.Loader, + loader: unsized(o.Loader), batchLoader: o.BatchLoader, } if cr.loader == nil && cr.batchLoader != nil { - cr.loader = cr.loadAsBatch + cr.loader = unsized(cr.loadAsBatch) } cr.getRefresher = cr.refresherWith(cr.loader) @@ -334,6 +334,13 @@ func (c *Cache[K, V]) SetTTL(key K, value V, ttl time.Duration) error { return c.core.setValue(key, value, c.core.valueCost(value), ttl, c.core.refreshAfter, refreshCache) } +// SetWithCost is Set with the value's cost given instead of taken from +// Options.Cost, for a value whose size is known only where it was produced. A +// negative cost counts as zero. +func (c *Cache[K, V]) SetWithCost(key K, value V, cost int64) error { + return c.core.setValue(key, value, max(cost, 0), c.core.ttl, c.core.refreshAfter, refreshCache) +} + // SetNegative records that the upstream reports no such key, for the configured // NegativeTTL. A successful write supersedes outstanding loads. func (c *Cache[K, V]) SetNegative(key K) error { @@ -525,7 +532,7 @@ func (c *core[K, V]) countLookup(s *shard[K, V], st Status, wasExpired bool) { // refresherWith is who a read is when it would refresh with loader: the cache, // or nobody when refresh is off or there is nothing to load with. -func (c *core[K, V]) refresherWith(loader func(context.Context, K) (V, error)) refresher { +func (c *core[K, V]) refresherWith(loader loadFunc[K, V]) refresher { if c.refreshAfter > 0 && loader != nil { return refreshCache } diff --git a/cost_test.go b/cost_test.go new file mode 100644 index 0000000..c1afe6d --- /dev/null +++ b/cost_test.go @@ -0,0 +1,91 @@ +package sanecache + +import ( + "context" + "errors" + "testing" + "time" +) + +func TestSetWithCost(t *testing.T) { + c := New(Options[string, string]{MaxBytes: 100, Cost: func(string) int64 { return 1 }, DisableCleanup: true}) + defer c.Close() + + if err := c.SetWithCost("k", "v", 60); err != nil || c.Bytes() != 60 { + t.Fatalf("err %v, bytes %d", err, c.Bytes()) + } + if err := c.SetWithCost("big", "v", 101); !errors.Is(err, ErrTooLarge) { + t.Fatalf("over budget: %v", err) + } + if err := c.SetWithCost("k", "v", -5); err != nil || c.Bytes() != 0 { + t.Fatalf("negative cost: err %v, bytes %d", err, c.Bytes()) + } + + a := New(Options[string, any]{MaxBytes: 100, Cost: func(any) int64 { return 1 }, DisableCleanup: true}) + defer a.Close() + if err := NewView(a, ViewOptions[int]{Name: "n"}).SetWithCost("k", 1, 7); err != nil || a.Bytes() != 7 { + t.Fatalf("view: err %v, bytes %d", err, a.Bytes()) + } + if err := NewView(nil, ViewOptions[int]{Name: "off"}).SetWithCost("k", 1, 7); !errors.Is(err, ErrDisabled) { + t.Fatalf("disabled view: %v", err) + } +} + +func TestGetOrLoadFuncWithCost(t *testing.T) { + c := New(Options[string, string]{ + TTL: refreshTTL, NegativeTTL: time.Minute, RefreshAfter: refreshAfter, + MaxBytes: 1000, Cost: func(string) int64 { return 1 }, DisableCleanup: true, + }) + defer c.Close() + advance := manualClock(c) + ctx := context.Background() + + cost := int64(40) + load := func(context.Context) (string, int64, error) { return "v", cost, nil } + if v, err := c.GetOrLoadFuncWithCost(ctx, "k", load); v != "v" || err != nil || c.Bytes() != 40 { + t.Fatalf("got %q, %v, bytes %d", v, err, c.Bytes()) + } + + // A refresh started by this method charges what its load reports, too. + cost = 70 + advance(refreshAfter + time.Second) + _, _ = c.GetOrLoadFuncWithCost(ctx, "k", load) + waitFor(t, "the refresh", func() bool { return c.Bytes() == 70 }) + + if v, err := c.GetOrLoadFuncWithCost(ctx, "big", func(context.Context) (string, int64, error) { + return "huge", 1001, nil + }); v != "huge" || err != nil || c.Stats().Rejections != 1 { + t.Fatalf("too large: %q, %v, %+v", v, err, c.Stats()) + } + if _, err := c.GetOrLoadFuncWithCost(ctx, "gone", func(context.Context) (string, int64, error) { + return "", 0, ErrNotFound + }); !errors.Is(err, ErrNotFound) { + t.Fatalf("not found: %v", err) + } + if _, st := c.Lookup("gone"); st != StatusNegative { + t.Fatalf("not found was not remembered: %v", st) + } + if _, err := c.GetOrLoadFuncWithCost(ctx, "k", nil); !errors.Is(err, ErrNoLoader) { + t.Fatalf("nil load: %v", err) + } +} + +func TestViewGetOrLoadFuncWithCost(t *testing.T) { + c := New(Options[string, any]{TTL: time.Minute, MaxBytes: 100, Cost: func(any) int64 { return 1 }, DisableCleanup: true}) + defer c.Close() + ctx := context.Background() + load := func(context.Context) (int, int64, error) { return 7, 30, nil } + + v := NewView(c, ViewOptions[int]{Name: "n", Cost: func(int) int64 { return 2 }}) + if got, err := v.GetOrLoadFuncWithCost(ctx, "k", load); got != 7 || err != nil || c.Bytes() != 30 { + t.Fatalf("got %d, %v, bytes %d", got, err, c.Bytes()) + } + if _, err := v.GetOrLoadFuncWithCost(ctx, "k", nil); !errors.Is(err, ErrNoLoader) { + t.Fatalf("nil load: %v", err) + } + + off := NewView(nil, ViewOptions[int]{Name: "off"}) + if got, err := off.GetOrLoadFuncWithCost(ctx, "k", load); got != 7 || err != nil { + t.Fatalf("disabled view: %d, %v", got, err) + } +} diff --git a/loader.go b/loader.go index d987286..18b8e73 100644 --- a/loader.go +++ b/loader.go @@ -24,11 +24,7 @@ func (c *Cache[K, V]) GetOrLoad(ctx context.Context, key K) (V, error) { return zero, ErrNoLoader } - if v, done, err := c.cached(ctx, key, c.core.loader); done { - return v, err - } - - return c.core.load(ctx, key, c.core.loader) + return c.getOrLoad(ctx, key, c.core.loader) } // GetOrLoadFunc is GetOrLoad with the loader passed in, for upstream calls that @@ -41,7 +37,33 @@ func (c *Cache[K, V]) GetOrLoadFunc(ctx context.Context, key K, load func(contex return zero, ErrNoLoader } - loader := func(ctx context.Context, _ K) (V, error) { return load(ctx) } + return c.getOrLoad(ctx, key, func(ctx context.Context, _ K) (V, int64, error) { + v, err := load(ctx) + + return v, -1, err + }) +} + +// GetOrLoadFuncWithCost is GetOrLoadFunc for a load that reports the cost of +// what it returns, which is used instead of Options.Cost, for values whose size +// is known only where they are produced. A negative cost counts as zero. +func (c *Cache[K, V]) GetOrLoadFuncWithCost( + ctx context.Context, key K, load func(context.Context) (V, int64, error), +) (V, error) { + if load == nil { + var zero V + + return zero, ErrNoLoader + } + + return c.getOrLoad(ctx, key, func(ctx context.Context, _ K) (V, int64, error) { + v, cost, err := load(ctx) + + return v, max(cost, 0), err + }) +} + +func (c *Cache[K, V]) getOrLoad(ctx context.Context, key K, loader loadFunc[K, V]) (V, error) { if v, done, err := c.cached(ctx, key, loader); done { return v, err } @@ -49,9 +71,26 @@ func (c *Cache[K, V]) GetOrLoadFunc(ctx context.Context, key K, load func(contex return c.core.load(ctx, key, loader) } +// loadFunc is a loader that also reports the cost of what it loaded, or -1 for +// the cost to come from the Cost function. +type loadFunc[K comparable, V any] func(context.Context, K) (V, int64, error) + +// unsized turns a loader from the options into a loadFunc, keeping nil nil. +func unsized[K comparable, V any](load func(context.Context, K) (V, error)) loadFunc[K, V] { + if load == nil { + return nil + } + + return func(ctx context.Context, key K) (V, int64, error) { + v, err := load(ctx, key) + + return v, -1, err + } +} + // cached answers a load from the cache when it can, starting the refresh it // claims with loader. It reports false only for a miss the caller can wait for. -func (c *Cache[K, V]) cached(ctx context.Context, key K, loader func(context.Context, K) (V, error)) (V, bool, error) { +func (c *Cache[K, V]) cached(ctx context.Context, key K, loader loadFunc[K, V]) (V, bool, error) { v, st, _, refresh := c.core.lookup(key, c.core.refresherWith(loader)) if refresh { c.core.refresh(ctx, key, loader) @@ -132,7 +171,7 @@ func (g *flightGroup[K, V]) drop(key K, cl *call[V]) { } } -func (c *core[K, V]) load(ctx context.Context, key K, loader func(context.Context, K) (V, error)) (V, error) { +func (c *core[K, V]) load(ctx context.Context, key K, loader loadFunc[K, V]) (V, error) { idx := c.shardIndex(key) g, s := c.flights[idx], c.shards[idx] @@ -173,7 +212,7 @@ func (c *core[K, V]) load(ctx context.Context, key K, loader func(context.Contex // refresh reloads key in the background ahead of its expiry, as an ordinary // flight that callers may join, unless a load of the key is already running. -func (c *core[K, V]) refresh(ctx context.Context, key K, loader func(context.Context, K) (V, error)) { +func (c *core[K, V]) refresh(ctx context.Context, key K, loader loadFunc[K, V]) { idx := c.shardIndex(key) g, s := c.flights[idx], c.shards[idx] @@ -201,14 +240,17 @@ func (c *core[K, V]) countCoalesced(s *shard[K, V]) { // a refresh. func (c *core[K, V]) run( ctx context.Context, g *flightGroup[K, V], s *shard[K, V], key K, cl *call[V], - loader func(context.Context, K) (V, error), refresh bool, + loader loadFunc[K, V], refresh bool, ) { defer s.endLoad(key, cl.token) g.run(key, cl, func() (V, error) { - v, err := loader(ctx, key) + v, cost, err := loader(ctx, key) + if cost < 0 { + cost = c.valueCost(v) + } switch { case err == nil: - _ = c.setValue(key, v, c.valueCost(v), c.ttl, c.refreshAfter, refreshCache, cl.token) + _ = c.setValue(key, v, cost, c.ttl, c.refreshAfter, refreshCache, cl.token) case errors.Is(err, ErrNotFound): _ = c.setNegative(key, c.negativeTTL, cl.token) } diff --git a/refresh_test.go b/refresh_test.go index e1a252a..db293ca 100644 --- a/refresh_test.go +++ b/refresh_test.go @@ -527,7 +527,7 @@ func TestRefreshLeavesARunningLoadToPublish(t *testing.T) { defer c.Close() g := c.core.flights[c.core.shardIndex("k")] cl := running(g, "k") - c.core.refresh(context.Background(), "k", never) + c.core.refresh(context.Background(), "k", unsized(never)) if g.calls["k"] != cl { t.Fatal("refresh replaced the running load") } @@ -550,7 +550,7 @@ func TestRefreshLeavesARunningLoadToPublish(t *testing.T) { v := NewView(vc, ViewOptions[int]{Name: "v", RefreshAfter: refreshAfter, Loader: never}) vg := v.flights[vc.core.shardIndex("v:k")] vcl := running(vg, "k") - v.refresh(context.Background(), "k", never) + v.refresh(context.Background(), "k", unsized(never)) if vg.calls["k"] != vcl { t.Fatal("a view refresh replaced the running load") } diff --git a/view.go b/view.go index 23c0de1..e5ad9ed 100644 --- a/view.go +++ b/view.go @@ -61,7 +61,7 @@ type View[T any] struct { // getRefresher is refresherWith(loader), worked out once. getRefresher refresher - loader func(context.Context, string) (T, error) + loader loadFunc[string, T] flights []*flightGroup[string, T] // stats is shared by every view of this name on this cache. @@ -89,7 +89,7 @@ func NewView[T any](c *Cache[string, any], o ViewOptions[T]) *View[T] { if c == nil { v := &View[T]{ - loader: o.Loader, + loader: unsized(o.Loader), name: o.Name, prefix: o.Name + viewSeparator, countStats: true, @@ -102,7 +102,7 @@ func NewView[T any](c *Cache[string, any], o ViewOptions[T]) *View[T] { v := &View[T]{ cache: c, - loader: o.Loader, + loader: unsized(o.Loader), name: o.Name, prefix: o.Name + viewSeparator, cost: o.Cost, @@ -122,7 +122,7 @@ func NewView[T any](c *Cache[string, any], o ViewOptions[T]) *View[T] { o.RefreshAfter, v.ttl)) } v.refreshAfter = o.RefreshAfter - v.getRefresher = v.refresherWith(o.Loader) + v.getRefresher = v.refresherWith(v.loader) // Made with or without a view loader: GetOrLoadFunc brings its own. v.flights = make([]*flightGroup[string, T], len(c.core.shards)) @@ -157,7 +157,7 @@ var background = context.Background() // lookup is Lookup as reader by, refreshing with loader. func (v *View[T]) lookup( - ctx context.Context, key string, by refresher, loader func(context.Context, string) (T, error), + ctx context.Context, key string, by refresher, loader loadFunc[string, T], ) (T, Status) { var zero T @@ -210,6 +210,15 @@ func (v *View[T]) SetTTL(key string, value T, ttl time.Duration) error { return v.cache.core.setValue(v.prefix+key, value, v.valueCost(value), ttl, v.refreshAfter, refreshView) } +// SetWithCost is Cache.SetWithCost for this view, with the view's TTL. +func (v *View[T]) SetWithCost(key string, value T, cost int64) error { + if v.cache == nil { + return ErrDisabled + } + + return v.cache.core.setValue(v.prefix+key, value, max(cost, 0), v.ttl, v.refreshAfter, refreshView) +} + // SetNegative records that the upstream reports no such key, for the view's // NegativeTTL. func (v *View[T]) SetNegative(key string) error { @@ -273,7 +282,7 @@ func (c *core[K, V]) viewCounters(name string) *counters { } // refresherWith is who a read of this view is when it would refresh with loader. -func (v *View[T]) refresherWith(loader func(context.Context, string) (T, error)) refresher { +func (v *View[T]) refresherWith(loader loadFunc[string, T]) refresher { if v.refreshAfter > 0 && loader != nil { return refreshView } diff --git a/view_loader.go b/view_loader.go index baa930d..175ff14 100644 --- a/view_loader.go +++ b/view_loader.go @@ -15,11 +15,7 @@ func (v *View[T]) GetOrLoad(ctx context.Context, key string) (T, error) { return zero, ErrNoLoader } - if val, done, err := v.cached(ctx, key, v.loader); done { - return val, err - } - - return v.loadWith(ctx, key, v.loader) + return v.getOrLoad(ctx, key, v.loader) } // GetOrLoadFunc is Cache.GetOrLoadFunc for this view, sharing loads with its @@ -30,22 +26,34 @@ func (v *View[T]) GetOrLoadFunc(ctx context.Context, key string, load func(conte return zero, ErrNoLoader } - loader := func(ctx context.Context, _ string) (T, error) { return load(ctx) } - if val, done, err := v.cached(ctx, key, loader); done { - return val, err - } + return v.getOrLoad(ctx, key, func(ctx context.Context, _ string) (T, int64, error) { + val, err := load(ctx) - return v.loadWith(ctx, key, loader) + return val, -1, err + }) } -// cached is Cache.cached for a view. -func (v *View[T]) cached(ctx context.Context, key string, loader func(context.Context, string) (T, error)) (T, bool, error) { - val, st := v.lookup(ctx, key, v.refresherWith(loader), loader) +// GetOrLoadFuncWithCost is Cache.GetOrLoadFuncWithCost for this view. +func (v *View[T]) GetOrLoadFuncWithCost( + ctx context.Context, key string, load func(context.Context) (T, int64, error), +) (T, error) { + if load == nil { + var zero T - return answered(ctx, val, st) + return zero, ErrNoLoader + } + + return v.getOrLoad(ctx, key, func(ctx context.Context, _ string) (T, int64, error) { + val, cost, err := load(ctx) + + return val, max(cost, 0), err + }) } -func (v *View[T]) loadWith(ctx context.Context, key string, loader func(context.Context, string) (T, error)) (T, error) { +func (v *View[T]) getOrLoad(ctx context.Context, key string, loader loadFunc[string, T]) (T, error) { + if val, done, err := v.cached(ctx, key, loader); done { + return val, err + } if v.cache == nil { return v.loadUncached(ctx, key, loader) } @@ -53,8 +61,15 @@ func (v *View[T]) loadWith(ctx context.Context, key string, loader func(context. return v.load(ctx, key, loader) } +// cached is Cache.cached for a view. +func (v *View[T]) cached(ctx context.Context, key string, loader loadFunc[string, T]) (T, bool, error) { + val, st := v.lookup(ctx, key, v.refresherWith(loader), loader) + + return answered(ctx, val, st) +} + // loadUncached is load for a view without a cache: shared, but not kept. -func (v *View[T]) loadUncached(ctx context.Context, key string, loader func(context.Context, string) (T, error)) (T, error) { +func (v *View[T]) loadUncached(ctx context.Context, key string, loader loadFunc[string, T]) (T, error) { g := v.flights[0] g.mu.Lock() @@ -66,13 +81,17 @@ func (v *View[T]) loadUncached(ctx context.Context, key string, loader func(cont return g.wait(ctx, key, cl) } cl := g.startAndUnlock(ctx, key, nil, func(ctx context.Context, cl *call[T]) { - g.run(key, cl, func() (T, error) { return loader(ctx, key) }, func(o loadOutcome) { v.stats.countLoad(o, false) }) + g.run(key, cl, func() (T, error) { + val, _, err := loader(ctx, key) + + return val, err + }, func(o loadOutcome) { v.stats.countLoad(o, false) }) }) return g.wait(ctx, key, cl) } -func (v *View[T]) load(ctx context.Context, key string, loader func(context.Context, string) (T, error)) (T, error) { +func (v *View[T]) load(ctx context.Context, key string, loader loadFunc[string, T]) (T, error) { c := v.cache.core fullKey := v.prefix + key idx := c.shardIndex(fullKey) @@ -114,7 +133,7 @@ func (v *View[T]) load(ctx context.Context, key string, loader func(context.Cont } // refresh is Cache's refresh for a view's key, with the view's loader. -func (v *View[T]) refresh(ctx context.Context, key string, loader func(context.Context, string) (T, error)) { +func (v *View[T]) refresh(ctx context.Context, key string, loader loadFunc[string, T]) { c := v.cache.core fullKey := v.prefix + key idx := c.shardIndex(fullKey) @@ -134,16 +153,19 @@ func (v *View[T]) refresh(ctx context.Context, key string, loader func(context.C // run is Cache's run for a view's key, counted for the view too. func (v *View[T]) run( ctx context.Context, g *flightGroup[string, T], s *shard[string, any], key string, cl *call[T], - loader func(context.Context, string) (T, error), refresh bool, + loader loadFunc[string, T], refresh bool, ) { c := v.cache.core fullKey := v.prefix + key defer s.endLoad(fullKey, cl.token) g.run(key, cl, func() (T, error) { - value, err := loader(ctx, key) + value, cost, err := loader(ctx, key) + if cost < 0 { + cost = v.valueCost(value) + } switch { case err == nil: - _ = c.setValue(fullKey, value, v.valueCost(value), v.ttl, v.refreshAfter, refreshView, cl.token) + _ = c.setValue(fullKey, value, cost, v.ttl, v.refreshAfter, refreshView, cl.token) case errors.Is(err, ErrNotFound): _ = c.setNegative(fullKey, v.negativeTTL, cl.token) } From c9748962efe4685e631be068493669512ce76a6e Mon Sep 17 00:00:00 2001 From: Andrey Ryabokon Date: Fri, 9 Oct 2026 08:32:11 +0300 Subject: [PATCH 7/9] Add Stats.Sub for exporters that send increments --- CHANGELOG.md | 3 +++ README.md | 3 ++- stats.go | 30 ++++++++++++++++++++++++++++++ stats_test.go | 41 +++++++++++++++++++++++++++++++++++++++++ view.go | 17 +++++++++++++++++ 5 files changed, 93 insertions(+), 1 deletion(-) create mode 100644 stats_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index e56f2e4..4fbd816 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,9 @@ While the major version is 0, the public API may change in any release. - `SetWithCost` and `GetOrLoadFuncWithCost` on `Cache` and `View`: the value's cost comes with it instead of from the `Cost` function, for values whose size is known only where they are produced, such as the length of the response they were decoded from. +- `Stats.Sub` and `ViewStats.Sub`: the counters gained since a previous snapshot, which is + what a metrics exporter sends each interval. A counter that went down, from a cache + created anew, comes back whole. ### Changed diff --git a/README.md b/README.md index fd2c2ab..9ced006 100644 --- a/README.md +++ b/README.md @@ -385,7 +385,8 @@ for name, vs := range c.ViewStats() { } ``` -Alert on `Rejections`: keys that can never be cached. `Coalesced` against `Loads` is what +`s.Sub(prev)` is what was gained since the previous snapshot, the increments a statsd +exporter sends. Alert on `Rejections`: keys that can never be cached. `Coalesced` against `Loads` is what single flight saves. `LoadErrors` leaves out `ErrNotFound`, which is in `LoadNotFound`, so ids that are gone do not look like failures. diff --git a/stats.go b/stats.go index b40f818..36e3ae2 100644 --- a/stats.go +++ b/stats.go @@ -43,6 +43,36 @@ func (s Stats) HitRate() float64 { return float64(s.Hits+s.Negatives) / float64(total) } +// Sub returns the counters gained since prev, which is what an exporter sends +// each interval. A counter below its value in prev means the cache was created +// anew, and is returned whole. Entries and Bytes are s's own. +func (s Stats) Sub(prev Stats) Stats { + for _, f := range []struct { + cur *int64 + prev int64 + }{ + {&s.Hits, prev.Hits}, {&s.Misses, prev.Misses}, {&s.Negatives, prev.Negatives}, + {&s.Evictions, prev.Evictions}, {&s.Expirations, prev.Expirations}, + {&s.Replacements, prev.Replacements}, {&s.Rejections, prev.Rejections}, + {&s.TypeMisses, prev.TypeMisses}, {&s.Loads, prev.Loads}, {&s.LoadNotFound, prev.LoadNotFound}, + {&s.LoadErrors, prev.LoadErrors}, {&s.Batches, prev.Batches}, {&s.Coalesced, prev.Coalesced}, + {&s.Refreshes, prev.Refreshes}, {&s.RefreshErrors, prev.RefreshErrors}, + } { + *f.cur = since(*f.cur, f.prev) + } + + return s +} + +// since is how much a counter grew from prev, or all of it if it was reset. +func since(cur, prev int64) int64 { + if cur < prev { + return cur + } + + return cur - prev +} + // addTo folds one shard's counters into a snapshot. func (c *counters) addTo(s *Stats) { s.Hits += c.hits.Load() diff --git a/stats_test.go b/stats_test.go new file mode 100644 index 0000000..c64ff7e --- /dev/null +++ b/stats_test.go @@ -0,0 +1,41 @@ +package sanecache + +import ( + "reflect" + "testing" +) + +// TestStatsSub covers every counter by reflection, so that a counter added +// later and forgotten in Sub fails here. +func TestStatsSub(t *testing.T) { + testSub(t, func(cur, prev Stats) Stats { return cur.Sub(prev) }) + testSub(t, func(cur, prev ViewStats) ViewStats { return cur.Sub(prev) }) +} + +func testSub[S any](t *testing.T, sub func(cur, prev S) S) { + t.Helper() + + var cur, prev, reset S + cv, pv, rv := reflect.ValueOf(&cur).Elem(), reflect.ValueOf(&prev).Elem(), reflect.ValueOf(&reset).Elem() + for i := range cv.NumField() { + v := int64(i + 1) + cv.Field(i).SetInt(10 * v) + pv.Field(i).SetInt(3 * v) + rv.Field(i).SetInt(100 * v) + } + + got, wasReset := reflect.ValueOf(sub(cur, prev)), reflect.ValueOf(sub(cur, reset)) + for i := range cv.NumField() { + name, v := cv.Type().Field(i).Name, int64(i+1) + wantDelta, wantReset := 7*v, 10*v + if name == "Entries" || name == "Bytes" { + wantDelta = 10 * v + } + if g := got.Field(i).Int(); g != wantDelta { + t.Errorf("%T.Sub: %s = %d, want %d", cur, name, g, wantDelta) + } + if g := wasReset.Field(i).Int(); g != wantReset { + t.Errorf("%T.Sub after a reset: %s = %d, want %d", cur, name, g, wantReset) + } + } +} diff --git a/view.go b/view.go index e5ad9ed..b3314be 100644 --- a/view.go +++ b/view.go @@ -323,6 +323,23 @@ type ViewStats struct { RefreshErrors int64 // refreshes that failed or panicked } +// Sub is Stats.Sub for a view. +func (s ViewStats) Sub(prev ViewStats) ViewStats { + for _, f := range []struct { + cur *int64 + prev int64 + }{ + {&s.Hits, prev.Hits}, {&s.Misses, prev.Misses}, {&s.Negatives, prev.Negatives}, + {&s.TypeMisses, prev.TypeMisses}, {&s.Loads, prev.Loads}, {&s.LoadNotFound, prev.LoadNotFound}, + {&s.LoadErrors, prev.LoadErrors}, {&s.Coalesced, prev.Coalesced}, + {&s.Refreshes, prev.Refreshes}, {&s.RefreshErrors, prev.RefreshErrors}, + } { + *f.cur = since(*f.cur, f.prev) + } + + return s +} + // HitRate reports hits as a fraction of all lookups, counting a cached negative // as a hit and a type miss as a miss. func (s ViewStats) HitRate() float64 { From 375b78dc00da0d747ee31084959f788e1fad4caf Mon Sep 17 00:00:00 2001 From: Andrey Ryabokon Date: Fri, 9 Oct 2026 08:33:36 +0300 Subject: [PATCH 8/9] Release v0.8.0 --- CHANGELOG.md | 10 ++++++++-- README.md | 4 ++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4fbd816..a63b53e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ While the major version is 0, the public API may change in any release. ## [Unreleased] +## [0.8.0] - 2026-10-09 + ### Added - `Options.RefreshAfter` and `ViewOptions.RefreshAfter`: a value read after it is this old @@ -32,7 +34,10 @@ While the major version is 0, the public API may change in any release. - The comparison benchmarks include theine and sturdyc, `make bench-compare` adds a serial `Get` and five runs of each, and the README tables are medians of those runs. - The README compares features with otter, theine, sturdyc and ttlcache, including what - they do that this library does not, and points to sturdyc for background refresh. + they do that this library does not. +- The `GetOrLoadFunc` and `GetManyOrLoad` examples had each other's descriptions on + pkg.go.dev. +- The README and the doc comments are shorter, with nothing taken out of what they promise. ## [0.7.0] - 2026-09-30 @@ -199,7 +204,8 @@ First cut. - Built-in counters via `Stats`, and an `OnEvict` callback carrying a reason. - `ErrTooLarge` from `Set` for values that can never fit. -[Unreleased]: https://github.com/andared/sanecache/compare/v0.7.0...HEAD +[Unreleased]: https://github.com/andared/sanecache/compare/v0.8.0...HEAD +[0.8.0]: https://github.com/andared/sanecache/compare/v0.7.0...v0.8.0 [0.7.0]: https://github.com/andared/sanecache/compare/v0.6.0...v0.7.0 [0.6.0]: https://github.com/andared/sanecache/compare/v0.5.0...v0.6.0 [0.5.0]: https://github.com/andared/sanecache/compare/v0.4.0...v0.5.0 diff --git a/README.md b/README.md index 9ced006..c6c5fd4 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ func main() { ```sh go mod init example.com/cache-demo -go get github.com/andared/sanecache@v0.7.0 +go get github.com/andared/sanecache@v0.8.0 go run . ``` @@ -483,6 +483,6 @@ against it, and value types with costs of their own ## Status -v0.7. The API above is what exists and is tested; expect it to move before v1. +v0.8. The API above is what exists and is tested; expect it to move before v1. Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for what this library optimises for. MIT licensed. From 37f8c1f5de2d50fa18e36f5b58e3e65d1f358dba Mon Sep 17 00:00:00 2001 From: Andrey Ryabokon Date: Fri, 9 Oct 2026 08:38:01 +0300 Subject: [PATCH 9/9] Pin the lint job to the Go version its linter can read --- .github/workflows/ci.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5e4225b..b8b40f5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -63,9 +63,11 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 + # Pinned with the linter, not stable: golangci-lint v2.13.2 cannot read the + # export data of Go 1.27.2. Move both together. - uses: actions/setup-go@v7 with: - go-version: stable + go-version: 1.27.1 cache: false - uses: golangci/golangci-lint-action@v9 with: