Right now analyze_results.py only computes descriptive statistics (mean, median, stddev, percentiles), but there's no way to tell whether a measured overhead difference is real or just noise. For example, if SELinux shows +3.2% overhead and AppArmor shows +2.8%, we can't say the 0.4% gap is meaningful because it could easily be hypervisor preemption or cache state randomness.
Two things need to be added:
-
95% Confidence Intervals on the overhead percentage, using bootstrap resampling. This gives a range like +3.2% (95% CI: +2.8% to +3.6%) instead of just a point estimate. If the CI for two MACs overlaps, we can't claim one is faster. If they don't overlap, we have statistical evidence. Bootstrap is the right method here because the overhead ratio (MAC − baseline) / baseline is a derived quantity that doesn't have a clean analytical CI formula.
-
Mann-Whitney U test instead of a t-test, because syscall latency distributions are right-skewed (many fast iterations, a few slow outliers from cache misses and preemption), not Gaussian. Mann-Whitney U is non-parametric—it doesn't assume any distribution shape. The test answers: "Is the MAC latency distribution statistically different from baseline?" with a p-value. If p < 0.05, the overhead is statistically significant; if not, we can't claim there's a real difference.
The output should look something like:
Category MAC Overhead 95% CI p-value Significant?
network selinux +3.2% [+2.8%, +3.6%] 0.0001 YES
network apparmor +2.8% [+2.1%, +3.5%] 0.0012 YES
ptrace apparmor +1.2% [-0.3%, +2.7%] 0.0823 NO
That last row is the important case: the CI crosses zero and p > 0.05, meaning the 1.2% overhead is not statistically distinguishable from noise. Without these tests, we'd wrongly report it as real overhead.
This requires adding SciPy as a dependency (pip install scipy) and modifying analyze_results.py to compute bootstrap confidence intervals and scipy.stats.mannwhitneyu for each MAC-vs-baseline pair, then include ci_lower, ci_upper, and p_value in the summary output and plots.
Right now
analyze_results.pyonly computes descriptive statistics (mean, median, stddev, percentiles), but there's no way to tell whether a measured overhead difference is real or just noise. For example, if SELinux shows +3.2% overhead and AppArmor shows +2.8%, we can't say the 0.4% gap is meaningful because it could easily be hypervisor preemption or cache state randomness.Two things need to be added:
95% Confidence Intervals on the overhead percentage, using bootstrap resampling. This gives a range like
+3.2% (95% CI: +2.8% to +3.6%)instead of just a point estimate. If the CI for two MACs overlaps, we can't claim one is faster. If they don't overlap, we have statistical evidence. Bootstrap is the right method here because the overhead ratio(MAC − baseline) / baselineis a derived quantity that doesn't have a clean analytical CI formula.Mann-Whitney U test instead of a t-test, because syscall latency distributions are right-skewed (many fast iterations, a few slow outliers from cache misses and preemption), not Gaussian. Mann-Whitney U is non-parametric—it doesn't assume any distribution shape. The test answers: "Is the MAC latency distribution statistically different from baseline?" with a p-value. If
p < 0.05, the overhead is statistically significant; if not, we can't claim there's a real difference.The output should look something like:
That last row is the important case: the CI crosses zero and
p > 0.05, meaning the 1.2% overhead is not statistically distinguishable from noise. Without these tests, we'd wrongly report it as real overhead.This requires adding SciPy as a dependency (
pip install scipy) and modifyinganalyze_results.pyto compute bootstrap confidence intervals andscipy.stats.mannwhitneyufor each MAC-vs-baseline pair, then includeci_lower,ci_upper, andp_valuein the summary output and plots.