diff --git a/tetragon/CFP-4191-per-workload-policy.md b/tetragon/CFP-4191-per-workload-policy.md index 772d188..d327abb 100644 --- a/tetragon/CFP-4191-per-workload-policy.md +++ b/tetragon/CFP-4191-per-workload-policy.md @@ -12,17 +12,25 @@ ## Summary -Today, with Tetragon it is not possible to share a common enforcement logic across many Kubernetes workloads. For each workload users need to create a separate `TracingPolicy` with the same enforcement but with different values. +Today, applying Tetragon policies for different Kubernetes (K8s) workloads requires a different policy for each workload (using the `podSelector` and `containerSelector` fields) and loads a different eBPF program for each policy. In each eBPF program, the eBPF code checks whether the executing process is subject to the corresponding policy. This approach introduces scalability limitations both from the UX perspective but also from the eBPF path perspective. The goal of this CFP is to explore solutions that address these limitations and propose a concrete solution to address them. + +This topic was initially discussed in https://github.com/cilium/tetragon/issues/4191. ## Motivation -The current approach of one `TracingPolicy` per workload leads to 2 main problems: +Let's consider the following use-case: creating per-workload policies that specify the subset of binaries each workload is allowed to execute. + +The current approach of one `TracingPolicy` per workload and a separate eBPF program per policy leads to two problems, which we discuss separately. + +### Scalability limitations of the eBPF implementation (P1) + +The current eBPF implementation (one program + many maps per `TracingPolicy`) scales poorly in clusters with many per-workload policies. First, certain hook types have an attachment limit of `BPF_MAX_TRAMP_LINKS` ([38 on x86](https://elixir.bootlin.com/linux/v6.14.11/source/include/linux/bpf.h#L1138)). This means that we cannot load more than 38 policies on the same hook of this type. Second, for each loaded program we [check](https://github.com/cilium/tetragon/blob/fdd7f014e4172d09f4fcc250f8a5790e764428f8/bpf/process/policy_filter.h#L51-L54) whether the policy applies to the given workload. This wastes a lot of CPU cycles, especially in cases where processes match a small subset of the existing policies. -- P1 (Scaling): The current eBPF implementation (one program + many maps per `TracingPolicy`) scales poorly in clusters with many per-workload policies (attachment limits, redundant evaluations, memory growth). +(Note(kkourt): is there an argument to be made for reducing memory footprint on the eBPF maps?) -- P2 (UX / Composition): Crafting many near-identical `TracingPolicy` resources with only small differences (e.g. filter values) is operationally cumbersome and error prone. +### Ergonomic (UX / Composition) Challenges (P2) -In this document we propose to tackle scaling (P1) and UX (P2) separately. +Crafting many near-identical `TracingPolicy` resources with only small differences (e.g. filter values) is operationally cumbersome, error prone, and makes the policies hard to maintain over time. ## Goals