T1: eBPF profiling capture of a live qfc-node validator

[lai3d]

The T1 evidence item from ROADMAP-SRE — kernel-level traces of a live validator, with the optimization targets the rest of the roadmap references. Captured on qfc-testnet-node (VPS-B, Ubuntu 24.04 / kernel 6.17 aarch64, EBS-backed nvme) against the running qfc-node-1 container.

Docs-only — traces, reproducible bpftrace programs + runner, and a findings note. No code changes.

Findings (production build staging-sha-8cf3cb0, steady-state testnet)

1. ~47% of on-CPU is BLAKE3 on the portable (scalar) backend on an aarch64 host — NEON SIMD isn't compiled in. About half of node CPU is recoverable hashing work. Highest-value optimization target; feeds a T3 follow-up.
2. ~17% on-CPU + ~20k write()/s is sync-protocol logging — SyncResponse formatted via tracing_subscriber::fmt, mirrored 1:1 by containerd-shim/dockerd (stdout → Docker json-log). Chain-side analogue of the Loki disk-exhaustion incident.
3. The deployed build does not fsync canonical blocks — only alloy fsyncs; qfc-node off-CPU is futex + tcp, no io_schedule. This is the baseline that T3.2 (PR T3.2: crash-atomic block commit + durability ADR #103 set_sync, merged-not-deployed) will change, and concretely why RPO=0 matters.
4. Disk is EBS — write service time 1–8 ms (tail 64 ms) dominates, not IOPS. Supports the T3.1 cache/bloom work (PR T3.1/T3.3: per-CF cache+bloom options, statistics hook, robustness fixes #105) and confirms a per-block fsync is affordable within the block interval.

Deliverables

• docs/profiling/T1-eBPF.md — findings note with the breakdown + cross-refs to T3.1/T3.2/T8.
• docs/profiling/ebpf/ — raw traces (01–05), 00-context.txt, and 05-oncpu.folded (flame-graph source; render with inferno-flamegraph/flamegraph.pl).
• scripts/profiling/ — capture.sh + four standalone .bt programs (no bcc dependency) to reproduce.

Notes

• No write-stall off-CPU flame (the roadmap's example artifact) exists in this build — finding 3 — so it's intentionally absent; re-run offcpu-qfcnode.bt post-T3.2-deploy to capture one.
• Steady-state testnet load; an under-stress capture (force compaction) on a dedicated node is the noted follow-up.
• Capture ran read-only/observability tracing; remote /tmp workspace cleaned up after.

🤖 Generated with Claude Code