EFAULT might have useful log that is not displayed

[GitSparTV]

Describe the bug

In some cases verifier rejects a program and instead logs with useful info shows this message:

load program: bad address (hit verifier bug, increase LogSizeStart to fit the log and check dmesg)

However, logs says this:

...
2192: (56) if w0 != 0x0 goto pc+373
The sequence of 8193 jumps is too complex.
verification time 850563 usec
stack depth 64+8+0+8+8+64+8+0+8+0+40+24+280+112+80+40+8+16+0+0+0+0+16+8+8+24+0+8+0+16+0+0+0+0+0+0
processed 336006 insns (limit 1000000) max_states_per_insn 19 total_states 13435 peak_states 2210 mark_read 177

Here is a simple but improper fix:

ebpf/prog.go

Lines 526 to 530 in 18b5c5b

	case errors.Is(err, unix.EFAULT):
	// EFAULT is returned when the kernel hits a verifier bug, and always
	// overrides ENOSPC, defeating the buffer growth strategy. Warn the user
	// that they may need to increase the buffer size manually.
	return nil, fmt.Errorf("load program: %w (hit verifier bug, increase LogSizeStart to fit the log and check dmesg)", err)

 		// EFAULT is returned when the kernel hits a verifier bug, and always
 		// overrides ENOSPC, defeating the buffer growth strategy. Warn the user
 		// that they may need to increase the buffer size manually.
-		return nil, fmt.Errorf("load program: %w (hit verifier bug, increase LogSizeStart to fit the log and check dmesg)", err)
+		return nil, internal.ErrorWithLog("load program (hit verifier bug, increase LogSizeStart to fit the log and check dmesg)", err, logBuf)
 
 	case errors.Is(err, unix.EINVAL):
 		if bytes.Contains(tail, coreBadCall) {

It took several weeks of blind debugging when we could use at least some info with the logs

How to reproduce

Version information

main branch

[ti-mo]

@GitSparTV Thanks for the issue! As mentioned in the comment, the reasons for this behaviour are two-fold. First the (historical?) difficulty of forcing the kernel to return EFAULT, since it always[*] means there's a verifier bug, which needs to be fixed in the kernel, not the library. This makes writing robust tests impossible. Secondly, under these circumstances it's highly unlikely we'll be able to return anything useful, since the verifier log would be truncated after the first few lines, since the auto-growth strategy wouldn't work.

[*] I don't know which kernel version you're on, but you were unfortunate to be affected by an inconsistency/bug in return code handling. I assume this error occurred in a bpf2bpf function given there's stack depth info? Looks like Anton fixed this here: https://lore.kernel.org/all/20251019202145.3944697-2-a.s.protopopov@gmail.com/. This should have shipped in the first 7.0 rc. If you're in a position to run a mainline kernel, please give this a try. I can't really tell from the patches when this could've started to occur.

As such, I'm hesitant to merge this, since we can't write a robust test for it. On the other hand, If there's anything in the log, we could technically display it as long as the warning is clear. On second thought, I think we actually intend to display some verifier log output, but this likely got lost in a refactor. This code is really tricky and has seen a few rewrites throughout the years. Could you give some example output of what the error message looks like with your patch included?

[GitSparTV]

Thanks for the reply.
I'm contributing into a tool that supports kernels starting from 5.4. So, when I went to test my BPF code on older kernels (that was the moment I found Ubuntu already has 6+ kernel), I stumbled across uninformative line of a verifier bug.
Ok, nice to know, but I need to fix my code somehow for this version 😀 . Any more information than this could gave more insights nevertheless.

[ti-mo]

Ok, nice to know, but I need to fix my code somehow for this version 😀 . Any more information than this could gave more insights nevertheless.

Not sure I understand correctly, do you need more info to continue?

[GitSparTV]

Not sure I understand correctly, do you need more info to continue?

Yes, I could look at least some verifier log and look where he died. At least some instructions are mentioned there and I can figure out where to place additional guards or limit loops.

For example, we had a perfectly working debug build (it enables debug logs using macro), but on a release build this fail message was occurring. This of course not the cause, but we still had to figure out blindly without any sort of feedback from the verifier