btf: replace global BTF cache with user provided cache

[matthyx]

Problem

globalCache in btf/kernel.go holds strong *Spec pointers to the kernel and module BTF specs. Once loaded, these ~20 MiB of type information are never released, even after all eBPF programs have been loaded and no CollectionSpec retains a reference to them.

FlushKernelSpec() exists as a manual escape hatch but requires callers to know when it is safe to call — which is inherently racy in a library.

Solution

Change globalCache.kernel and globalCache.modules from strong *Spec pointers to weak.Pointer[Spec] (Go 1.24+). The GC will collect the cached specs automatically once no CollectionSpec (or other caller) holds a strong reference to them.

Key behaviours preserved:

• Amortisation: The cache still avoids re-parsing /sys/kernel/btf/vmlinux across multiple LoadKernelSpec calls that overlap in time.
• Lazy reload: If the GC collects the cached spec, the next LoadKernelSpec call transparently re-parses it.
• FlushKernelSpec(): Still works; zeroing a weak.Pointer is equivalent to the previous = nil. Now deprecated since explicit flushing is no longer needed.
• NewCache(): Handles the case where the weak pointer has already been collected, falling back to an empty Cache (which then loads on demand via Kernel()).

Cache is now self-contained

Previously, Cache.Module re-used the global module cache and then rebased the result against the Cache's own kernel decoder. That created a hidden invariant: the Cache's kernel decoder had to share its raw mmap'd buffer with whatever kernel spec the global cache currently held — otherwise rebaseDecoder would fail with "raw BTF differs".

Under strong references this invariant held permanently. Under weak references it could break: if the global cached kernel spec were GC'd between Cache.Kernel() and Cache.Module(), the next module load would parse a fresh kernel mmap and the rebase would blow up.

Cache.Module now loads module BTF directly against the Cache's own kernel via loadKernelModuleSpec(name, c.kernelTypes). The Cache no longer depends on the global cache's state for correctness, and no kernelSource anchor field is needed. Memory reclaim is unchanged: the mmap'd vmlinux buffer is freed (munmap cleanup on sharedBuf) once the last Cache (and any other strong holder) is dropped.

The trade-off is that a module loaded via Cache.Module is no longer re-cached in the global module cache, so a subsequent LoadKernelModuleSpec for the same module reparses sysfs. Module BTF is small (kilobytes) and this only matters if Cache and the package-level loaders are mixed for the same modules.

Memory impact

In a long-running process that loads a handful of eBPF programs at startup and then runs indefinitely, this frees ~20 MiB of Go heap automatically — without any caller changes.

Measured in a Kubernetes node-agent workload (inspektor-gadget) where kernel BTF was the dominant post-load heap consumer.

Go version requirement

weak.Pointer was stabilised in Go 1.24. The module already requires Go 1.23; this PR bumps the minimum to Go 1.24.

Testing

Existing TestLoadKernelSpec and TestCache pass. The cache correctness is unchanged — the weak pointer is semantically transparent to callers as long as they hold a strong reference (via a returned *Spec) across the critical section.

[ti-mo]

Thanks! Left some comments. Looks like there's still a failure in the selftests.

[matthyx]

@ti-mo looks like the CI passes now

[matthyx]

@ti-mo can you check my proposal on the module BTF?

[ti-mo]

@matthyx I've given this a lot of thought, but something tells me this isn't the right approach. Turning these into weak pointers completely defeats their purpose as a long-lived cache, since there's no guarantee whatsoever that the global cache will outlive two sequential calls to LoadCollectionSpec, at which point we may as well remove it since btf.Cache fills intra-load caching now. I understand the difficulty in finding a good point to call Flush(); I assume you do this after every collection load to keep consumption down.

Though I have one question: what does that 20MiB of heap contain? We have lazy decoding now, so we should only (mostly) inflate types that get returned to the caller. I don't know your use case, but 20MiB feels like a lot. Are you counting heap or RSS?

It's worth noting that a few things changed since introducing the global cache. Originally, it intended to amortize the cost of parsing BTF across multiple programs within a single LoadCollectionSpec, but the fact that it persisted across multiple collection loads was a nice bonus. Now that decoding is lazy + mmap-backed, a lot of that cost is gone. Conversely, I find the lower-level logic around how the caches interact almost impossible to reason about today.

With btf.Cache in the picture, it may be time to sunset the global cache altogether and allow the caller to pass one in through CollectionOptions. This way, the caller can decide for themselves when and how to manage a cache and we sidestep this issue altogether.

[matthyx]

Thanks, this makes sense. You're right that weak pointers turn the global cache from a deterministic long-lived cache into a best-effort memoization layer, which is probably the wrong trade-off here.

Given that we already use a btf.Cache during collection load, surfacing that through CollectionOptions seems like the cleaner direction. That lets callers choose the cache lifetime explicitly instead of the library trying to guess a safe flush
point.

The memory saving is heap, but more 17 than 20, you can see some info here: kubescape/node-agent#786 (comment)

Should I close this?

[ti-mo]

Hi, are you generating these replies using an LLM? If you're up for making the CollectionOptions change, feel free to reuse this PR for that.

[matthyx]

@ti-mo yes I use one to redact sentences from bullet points, as I'm not native speaker in English. Sorry if that offended you.

[matthyx]

@ti-mo something like that?

[dylandreimerink]

I think this updated PR is nice, having less memory overhead by default, and allowing power users to enable caching between collections.

[ti-mo]

Thanks! This cleans up nicely. Left a few nits, and we should prob make btf.Cache concurrency-safe.

[ti-mo]

We should simply remove this as a call to action to anyone that depended on this behaviour. For anyone relying on package-level caching, Flush() becoming a no-op is already sort-of a breaking change. We may as well make it explicit.

[ti-mo]

Nit: move this above LoadKernelModuleSpec and export instead?

[ti-mo]

Nit:

Suggested change

	// collection loads. Callers loading more than one collection should
	// Collection loads. Callers loading more than one Collection should

[ti-mo]

Nit:

Suggested change

	// collection loads.
	// Collection loads.

[ti-mo]

This looks straightforward enough to address, it would be a rather inconvenient limitation. Could you pick this up as well?

[matthyx]

@ti-mo updated