Wrong behavior on s390x

[alicefr]

We are currently facing an issue using this library on s390x. I tired to run the test TestProgramRun. I got:

go test -run TestProgramRun
--- FAIL: TestProgramRun (0.00s)
    prog_test.go:50:     0: LdXMemW dst: r2 src: r1 off: 4 imm: 0
             1: LdXMemW dst: r1 src: r1 off: 0 imm: 0
             2: MovReg dst: r3 src: r1
             3: AddImm dst: r3 imm: 14
             4: JGTReg dst: r3 off: -1 src: r2 <out>
             5: StMemB dst: r1 src: r0 off: 0 imm: 222
             6: StMemB dst: r1 src: r0 off: 1 imm: 173
             7: StMemB dst: r1 src: r0 off: 2 imm: 190
             8: StMemB dst: r1 src: r0 off: 3 imm: 239
        out:
             9: LdImmDW dst: r0 imm: 42
            11: Exit
        
    prog_test.go:59: can't load program: permission denied: 0: (61) r1 = *(u32 *)(r2 +4)
        R2 !read_ok
        processed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
FAIL
exit status 1
FAIL    github.com/cilium/ebpf  0.001s
Collapse

[lmb]

Hi!

That's probably because you're the first s390x user :) Note the following:

0: LdXMemW dst: r2 src: r1 off: 4 imm: 0

vs.

0: (61) r1 = *(u32 *)(r2 +4)

The operands are switched: the library thinks it's emitting r2 = *(u32 *)(r1 + 4), but the verifier tells us the opposite.

Can you give me some more information about your system (kernel, userspace)? Is it running in big endian mode?

[alicefr]

hi @lmb
:) yes, I guess we're the first who use this package on s390. So here the info:

$ cat /etc/os-release 
NAME=Fedora
VERSION="31 (Thirty One)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora 31 (Thirty One)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f31/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
$ uname -rm
5.3.7-301.fc31.s390x s390x

yes, it is running in bigendian mode. Probably, there's some endianness issue somewhere in the code. Having a look

[lmb]

Yeah, I think https://github.com/cilium/ebpf/blob/master/asm/instruction.go#L435-L445 needs to be swapped for big endian. Annoying, since the registers are encoded as an uint8, which binary.ByteOrder doesn't support natively.

[alicefr]

@lmb I tried a quick hack

git diff
diff --git a/asm/instruction.go b/asm/instruction.go
index 644de42..ae6f3a0 100644
--- a/asm/instruction.go
+++ b/asm/instruction.go
@@ -77,7 +77,7 @@ func (ins Instruction) Marshal(w io.Writer, bo binary.ByteOrder) (uint64, error)
 
        bpfi := bpfInstruction{
                ins.OpCode,
-               newBPFRegisters(ins.Dst, ins.Src),
+               newBPFRegisters(ins.Src, ins.Dst),
                ins.Offset,
                cons,
        }
$ go test -run TestProgramRun
PASS
ok  	github.com/cilium/ebpf	0.002s

and it worked. You were right!

[lmb]

Do all of the other tests pass on s390x?

[alicefr]

@lmb unfortunately not.

go test
--- FAIL: TestLoadCollectionSpec (0.00s)
    --- FAIL: TestLoadCollectionSpec/loader-clang-6.0.elf (0.00s)
        elf_reader_test.go:85: xdp_prog:
            	 0: MovImm dst: r1 imm: 0
            	 1: StXMemW dst: rfp src: r1 off: -4 imm: 0
            	 2: MovImm dst: r1 imm: 1
            	 3: StXMemW dst: rfp src: r1 off: -8 imm: 0
            	 4: MovImm dst: r1 imm: 2
            	 5: StXMemW dst: rfp src: r1 off: -12 imm: 0
            	 6: MovReg dst: r2 src: rfp
            	 7: AddImm dst: r2 imm: -4
            	 8: LoadMapPtr dst: r1 fd: -1 <hash_map>
            	10: Call FnMapLookupElem
            	11: MovReg dst: r2 src: rfp
            	12: AddImm dst: r2 imm: -8
            	13: LoadMapPtr dst: r1 fd: -1 <hash_map2>
            	15: Call FnMapLookupElem
            	16: MovReg dst: r2 src: rfp
            	17: AddImm dst: r2 imm: -12
            	18: LoadMapPtr dst: r1 fd: -1 <hash_map2>
            	20: Call FnMapLookupElem
            	21: MovImm dst: r1 imm: 1
            	22: Call -1 <global_fn>
            	23: AddImm dst: r0 imm: 1
            	24: Exit
            global_fn2:
            	25: MovReg dst: r0 src: r1
            	26: Exit
            global_fn:
            	27: MovReg dst: r6 src: r1
            	28: Call 10
            	29: MovReg dst: r7 src: r0
            	30: MovReg dst: r1 src: r6
            	31: Call -7
            	32: MovReg dst: r8 src: r0
            	33: AddReg dst: r8 src: r7
            	34: MovReg dst: r1 src: r6
            	35: Call -1 <global_fn3>
            	36: AddReg dst: r8 src: r0
            	37: MovReg dst: r0 src: r8
            	38: Exit
            static_fn:
            	39: MovReg dst: r0 src: r1
            	40: Exit
            global_fn3:
            	41: AddImm dst: r1 imm: 1
            	42: MovReg dst: r0 src: r1
            	43: Exit
            
        elf_reader_test.go:94: program xdp_prog: can't load program: errno 524: 0: (b7) r1 = 0
            1: (63) *(u32 *)(r10 -4) = r1
            2: (b7) r1 = 1
            3: (63) *(u32 *)(r10 -8) = r1
            4: (b7) r1 = 2
            5: (63) *(u32 *)(r10 -12) = r1
            6: (bf) r2 = r10
            7: (07) r2 += -4
            8: (18) r1 = 0x1fbd42800
            10: (85) call bpf_map_lookup_elem#1
            11: (bf) r2 = r10
            12: (07) r2 += -8
            13: (18) r1 = 0x1fbd42000
            15: (85) call bpf_map_lookup_elem#1
            16: (bf) r2 = r10
            17: (07) r2 += -12
            18: (18) r1 = 0x1fbd42000
            20: (85) call bpf_map_lookup_elem#1
            21: (b7) r1 = 1
            22: (85) call pc+4
            caller:
             R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
            callee:
             frame1: R1_w=invP1 R10=fp0
            27: (bf) r6 = r1
            28: (85) call pc+10
            caller:
             frame1: R6_w=invP1 R10=fp0
            callee:
             frame2: R1_w=invP1 R10=fp0
            39: (bf) r0 = r1
            40: (95) exit
            returning from callee:
             frame2: R0_w=invP1 R1=invP1 R10=fp0
            to caller at 29:
             frame1: R0_w=invP1 R6=invP1 R10=fp0
            
            from 40 to 29: frame1: R0_w=invP1 R6=invP1 R10=fp0
            29: (bf) r7 = r0
            30: (bf) r1 = r6
            31: (85) call pc-7
            caller:
             frame1: R6=invP1 R7_w=invP1 R10=fp0
            callee:
             frame2: R1_w=invP1 R10=fp0
            25: (bf) r0 = r1
            26: (95) exit
            returning from callee:
             frame2: R0_w=invP1 R1_w=invP1 R10=fp0
            to caller at 32:
             frame1: R0_w=invP1 R6=invP1 R7_w=invP1 R10=fp0
            
            from 26 to 32: frame1: R0_w=invP1 R6=invP1 R7_w=invP1 R10=fp0
            32: (bf) r8 = r0
            33: (0f) r8 += r7
            34: (bf) r1 = r6
            35: (85) call pc+5
            caller:
             frame1: R6=invP1 R7=invP1 R8=invP2 R10=fp0
            callee:
             frame2: R1=invP1 R10=fp0
            41: (07) r1 += 1
            42: (bf) r0 = r1
            43: (95) exit
            returning from callee:
             frame2: R0_w=invP2 R1_w=invP2 R10=fp0
            to caller at 36:
             frame1: R0_w=invP2 R6=invP1 R7=invP1 R8=invP2 R10=fp0
            
            from 43 to 36: frame1: R0_w=invP2 R6=invP1 R7=invP1 R8=invP2 R10=fp0
            36: (0f) r8 += r0
            37: (bf) r0 = r8
            38: (95) exit
            returning from callee:
             frame1: R0_w=invP4 R6=invP1 R7=invP1 R8_w=invP4 R10=fp0
            to caller at 23:
             R0_w=invP4 R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
            
            from 38 to 23: R0_w=invP4 R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
            23: (07) r0 += 1
            24: (95) exit
            JIT doesn't support bpf-to-bpf calls
            processed 41 insns (limit 1000000) max_states_per_insn 0 total_states 3 peak_states 3 mark_read 1
    --- FAIL: TestLoadCollectionSpec/loader-clang-7.elf (0.00s)
        elf_reader_test.go:85: xdp_prog:
            	 0: MovImm dst: r1 imm: 0
            	 1: StXMemW dst: rfp src: r1 off: -4 imm: 0
            	 2: MovImm dst: r1 imm: 1
            	 3: StXMemW dst: rfp src: r1 off: -8 imm: 0
            	 4: MovImm dst: r1 imm: 2
            	 5: StXMemW dst: rfp src: r1 off: -12 imm: 0
            	 6: MovReg dst: r2 src: rfp
            	 7: AddImm dst: r2 imm: -4
            	 8: LoadMapPtr dst: r1 fd: -1 <hash_map>
            	10: Call FnMapLookupElem
            	11: MovReg dst: r2 src: rfp
            	12: AddImm dst: r2 imm: -8
            	13: LoadMapPtr dst: r1 fd: -1 <hash_map2>
            	15: Call FnMapLookupElem
            	16: MovReg dst: r2 src: rfp
            	17: AddImm dst: r2 imm: -12
            	18: LoadMapPtr dst: r1 fd: -1 <hash_map2>
            	20: Call FnMapLookupElem
            	21: MovImm dst: r1 imm: 1
            	22: Call -1 <global_fn>
            	23: AddImm dst: r0 imm: 1
            	24: Exit
            global_fn2:
            	25: MovReg dst: r0 src: r1
            	26: Exit
            global_fn:
            	27: MovReg dst: r6 src: r1
            	28: Call 10
            	29: MovReg dst: r7 src: r0
            	30: MovReg dst: r1 src: r6
            	31: Call -7
            	32: MovReg dst: r8 src: r0
            	33: AddReg dst: r8 src: r7
            	34: MovReg dst: r1 src: r6
            	35: Call -1 <global_fn3>
            	36: AddReg dst: r8 src: r0
            	37: MovReg dst: r0 src: r8
            	38: Exit
            static_fn:
            	39: MovReg dst: r0 src: r1
            	40: Exit
            global_fn3:
            	41: MovReg dst: r0 src: r1
            	42: AddImm dst: r0 imm: 1
            	43: Exit
            
        elf_reader_test.go:94: program xdp_prog: can't load program: errno 524: 0: (b7) r1 = 0
            1: (63) *(u32 *)(r10 -4) = r1
            2: (b7) r1 = 1
            3: (63) *(u32 *)(r10 -8) = r1
            4: (b7) r1 = 2
            5: (63) *(u32 *)(r10 -12) = r1
            6: (bf) r2 = r10
            7: (07) r2 += -4
            8: (18) r1 = 0x1f84fe800
            10: (85) call bpf_map_lookup_elem#1
            11: (bf) r2 = r10
            12: (07) r2 += -8
            13: (18) r1 = 0x1f84f9000
            15: (85) call bpf_map_lookup_elem#1
            16: (bf) r2 = r10
            17: (07) r2 += -12
            18: (18) r1 = 0x1f84f9000
            20: (85) call bpf_map_lookup_elem#1
            21: (b7) r1 = 1
            22: (85) call pc+4
            caller:
             R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
            callee:
             frame1: R1_w=invP1 R10=fp0
            27: (bf) r6 = r1
            28: (85) call pc+10
            caller:
             frame1: R6_w=invP1 R10=fp0
            callee:
             frame2: R1_w=invP1 R10=fp0
            39: (bf) r0 = r1
            40: (95) exit
            returning from callee:
             frame2: R0_w=invP1 R1=invP1 R10=fp0
            to caller at 29:
             frame1: R0_w=invP1 R6=invP1 R10=fp0
            
            from 40 to 29: frame1: R0_w=invP1 R6=invP1 R10=fp0
            29: (bf) r7 = r0
            30: (bf) r1 = r6
            31: (85) call pc-7
            caller:
             frame1: R6=invP1 R7_w=invP1 R10=fp0
            callee:
             frame2: R1_w=invP1 R10=fp0
            25: (bf) r0 = r1
            26: (95) exit
            returning from callee:
             frame2: R0_w=invP1 R1_w=invP1 R10=fp0
            to caller at 32:
             frame1: R0_w=invP1 R6=invP1 R7_w=invP1 R10=fp0
            
            from 26 to 32: frame1: R0_w=invP1 R6=invP1 R7_w=invP1 R10=fp0
            32: (bf) r8 = r0
            33: (0f) r8 += r7
            34: (bf) r1 = r6
            35: (85) call pc+5
            caller:
             frame1: R6=invP1 R7=invP1 R8=invP2 R10=fp0
            callee:
             frame2: R1=invP1 R10=fp0
            41: (bf) r0 = r1
            42: (07) r0 += 1
            43: (95) exit
            returning from callee:
             frame2: R0_w=invP2 R1=invP1 R10=fp0
            to caller at 36:
             frame1: R0_w=invP2 R6=invP1 R7=invP1 R8=invP2 R10=fp0
            
            from 43 to 36: frame1: R0_w=invP2 R6=invP1 R7=invP1 R8=invP2 R10=fp0
            36: (0f) r8 += r0
            37: (bf) r0 = r8
            38: (95) exit
            returning from callee:
             frame1: R0_w=invP4 R6=invP1 R7=invP1 R8_w=invP4 R10=fp0
            to caller at 23:
             R0_w=invP4 R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
            
            from 38 to 23: R0_w=invP4 R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
            23: (07) r0 += 1
            24: (95) exit
            JIT doesn't support bpf-to-bpf calls
            processed 41 insns (limit 1000000) max_states_per_insn 0 total_states 3 peak_states 3 mark_read 1
    --- FAIL: TestLoadCollectionSpec/loader-clang-8.elf (0.00s)
        elf_reader_test.go:85: xdp_prog:
            	 0: MovImm dst: r1 imm: 0
            	 1: StXMemW dst: rfp src: r1 off: -4 imm: 0
            	 2: MovImm dst: r1 imm: 1
            	 3: StXMemW dst: rfp src: r1 off: -8 imm: 0
            	 4: MovImm dst: r1 imm: 2
            	 5: StXMemW dst: rfp src: r1 off: -12 imm: 0
            	 6: MovReg dst: r2 src: rfp
            	 7: AddImm dst: r2 imm: -4
            	 8: LoadMapPtr dst: r1 fd: -1 <hash_map>
            	10: Call FnMapLookupElem
            	11: MovReg dst: r2 src: rfp
            	12: AddImm dst: r2 imm: -8
            	13: LoadMapPtr dst: r1 fd: -1 <hash_map2>
            	15: Call FnMapLookupElem
            	16: MovReg dst: r2 src: rfp
            	17: AddImm dst: r2 imm: -12
            	18: LoadMapPtr dst: r1 fd: -1 <hash_map2>
            	20: Call FnMapLookupElem
            	21: MovImm dst: r1 imm: 1
            	22: Call -1 <global_fn>
            	23: AddImm dst: r0 imm: 1
            	24: Exit
            global_fn2:
            	25: MovReg dst: r0 src: r1
            	26: Exit
            global_fn:
            	27: MovReg dst: r6 src: r1
            	28: Call 10
            	29: MovReg dst: r7 src: r0
            	30: MovReg dst: r1 src: r6
            	31: Call -7
            	32: MovReg dst: r8 src: r0
            	33: AddReg dst: r8 src: r7
            	34: MovReg dst: r1 src: r6
            	35: Call -1 <global_fn3>
            	36: AddReg dst: r8 src: r0
            	37: MovReg dst: r0 src: r8
            	38: Exit
            static_fn:
            	39: MovReg dst: r0 src: r1
            	40: Exit
            global_fn3:
            	41: MovReg dst: r0 src: r1
            	42: AddImm dst: r0 imm: 1
            	43: Exit
            
        elf_reader_test.go:94: invalid argument: magic: 0xeb9f
            version: 1
            flags: 0x0
            hdr_len: 24
            type_off: 0
            type_len: 424
            str_off: 424
            str_len: 887
            btf_total_size: 1335
            [1] TYPEDEF uint32_t type_id=2
            [2] INT unsigned int Invalid int_data:20000000
    --- FAIL: TestLoadCollectionSpec/loader-clang-9.elf (0.00s)
        elf_reader_test.go:85: xdp_prog:
            	 0: LoadMapPtr dst: r1 fd: -1 <hash_map>
            	 2: LoadMapValue dst: r2, fd: -1 off: 0 <.bss>
            	 4: Call FnMapLookupElem
            	 5: LoadMapPtr dst: r1 fd: -1 <hash_map2>
            	 7: LoadMapValue dst: r2, fd: -1 off: 0 <.data>
            	 9: Call FnMapLookupElem
            	10: LoadMapPtr dst: r1 fd: -1 <hash_map2>
            	12: LoadMapValue dst: r2, fd: -1 off: 0 <.rodata>
            	14: Call FnMapLookupElem
            	15: LoadMapValue dst: r7, fd: -1 off: 4 <.rodata>
            	17: LdXMemW dst: r1 src: r7 off: 0 imm: 0
            	18: Call -1 <static_fn>
            	19: MovReg dst: r6 src: r0
            	20: LdXMemW dst: r1 src: r7 off: 0 imm: 0
            	21: Call -1 <global_fn>
            	22: AddReg dst: r0 src: r6
            	23: Exit
            global_fn2:
            	24: MovReg dst: r0 src: r1
            	25: Exit
            global_fn:
            	26: MovReg dst: r6 src: r1
            	27: Call 10
            	28: MovReg dst: r7 src: r0
            	29: MovReg dst: r1 src: r6
            	30: Call -7
            	31: MovReg dst: r8 src: r0
            	32: AddReg dst: r8 src: r7
            	33: MovReg dst: r1 src: r6
            	34: Call -1 <global_fn3>
            	35: AddReg dst: r8 src: r0
            	36: MovReg dst: r0 src: r8
            	37: Exit
            static_fn:
            	38: MovReg dst: r0 src: r1
            	39: Exit
            global_fn3:
            	40: MovReg dst: r0 src: r1
            	41: AddImm dst: r0 imm: 1
            	42: Exit
            
        elf_reader_test.go:94: invalid argument: magic: 0xeb9f
            version: 1
            flags: 0x0
            hdr_len: 24
            type_off: 0
            type_len: 868
            str_off: 868
            str_len: 916
            btf_total_size: 1808
            [1] STRUCT (anon) size=40 vlen=5
            	type type_id=2 bits_offset=0
            	key type_id=6 bits_offset=64
            	value type_id=6 bits_offset=128
            	max_entries type_id=2 bits_offset=192
            	map_flags type_id=2 bits_offset=256
            [2] PTR (anon) type_id=4
            [3] INT int Invalid int_data:20000001
--- FAIL: TestLink (0.00s)
    linker_test.go:43: 	0: MovReg dst: r0 src: r1
        	1: Call -1 <my_func>
        	2: Exit
        my_func:
        	3: Call -1 <my_other_func>
        	4: Exit
        my_other_func:
        	5: LdImmDW dst: r0 imm: 1337
        	7: Exit
        
    linker_test.go:49: can't load program: errno 524: 0: (bf) r0 = r1
        1: (85) call pc+1
        caller:
         R10=fp0
        callee:
         frame1: R1=ctx(id=0,off=0,imm=0) R10=fp0
        3: (85) call pc+1
        caller:
         frame1: R10=fp0
        callee:
         frame2: R1=ctx(id=0,off=0,imm=0) R10=fp0
        5: (18) r0 = 0x539
        7: (95) exit
        returning from callee:
         frame2: R0_w=invP1337 R1=ctx(id=0,off=0,imm=0) R10=fp0
        to caller at 4:
         frame1: R0_w=invP1337 R10=fp0
        
        from 7 to 4: frame1: R0_w=invP1337 R10=fp0
        4: (95) exit
        returning from callee:
         frame1: R0_w=invP1337 R10=fp0
        to caller at 2:
         R0_w=invP1337 R10=fp0
        
        from 4 to 2: R0_w=invP1337 R10=fp0
        2: (95) exit
        JIT doesn't support bpf-to-bpf calls
        processed 7 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
FAIL
exit status 1
FAIL	github.com/cilium/ebpf	0.161s

[alicefr]

But I'm not sure if those tests should be skipped as we get ENOTSUPP

[alicefr]

I tried to recompile the testdata:

$ clang --version
clang version 9.0.1 (Fedora 9.0.1-2.fc31)
Target: s390x-ibm-linux
Thread model: posix
InstalledDir: /usr/bin

$ cd testdata/
$ make clean
$ make loader-clang-9.elf rewrite.elf invalid_map.elf
$ ls
common.h  invalid_map.c  invalid_map.elf  loader.c  loader-clang-9.elf  Makefile  rewrite.c  rewrite.elf

Now, I get:

go test
--- FAIL: TestLoadCollectionSpec (0.00s)
    --- FAIL: TestLoadCollectionSpec/loader-clang-9.elf (0.00s)
        elf_reader_test.go:23: Can't parse ELF: file testdata/loader-clang-9.elf: load programs: program global_fn2: can't unmarshal instructions: offset 80: can't relocate instruction: call: global_fn3: incorrect source register
--- FAIL: TestLink (0.00s)
    linker_test.go:43: 	0: MovReg dst: r0 src: r1
        	1: Call -1 <my_func>
        	2: Exit
        my_func:
        	3: Call -1 <my_other_func>
        	4: Exit
        my_other_func:
        	5: LdImmDW dst: r0 imm: 1337
        	7: Exit
        
    linker_test.go:49: can't load program: errno 524: 0: (bf) r0 = r1
        1: (85) call pc+1
        caller:
         R10=fp0
        callee:
         frame1: R1=ctx(id=0,off=0,imm=0) R10=fp0
        3: (85) call pc+1
        caller:
         frame1: R10=fp0
        callee:
         frame2: R1=ctx(id=0,off=0,imm=0) R10=fp0
        5: (18) r0 = 0x539
        7: (95) exit
        returning from callee:
         frame2: R0_w=invP1337 R1=ctx(id=0,off=0,imm=0) R10=fp0
        to caller at 4:
         frame1: R0_w=invP1337 R10=fp0
        
        from 7 to 4: frame1: R0_w=invP1337 R10=fp0
        4: (95) exit
        returning from callee:
         frame1: R0_w=invP1337 R10=fp0
        to caller at 2:
         R0_w=invP1337 R10=fp0
        
        from 4 to 2: R0_w=invP1337 R10=fp0
        2: (95) exit
        JIT doesn't support bpf-to-bpf calls
        processed 7 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
FAIL
exit status 1
FAIL	github.com/cilium/ebpf	0.175s

[lmb]

Can you upload the compiled testdata somewhere? No idea how to get cross compiling to work for this.

Issues I can see:

• Decoding instructions from the ELF also needs to swap registers OR we need to reject ELF that are not in the native endian
• Something in BTF is also endian dependent
• The s390x JIT doesn't support function calls

We can't do much about the last point, except detect the issue and skip the test. Bummer, I think bpf-2-bpf calls are pretty important.

Is there a way to get access to a s390x VM with a recent kernel?

[alicefr]

@lmb I might request a VM for the community, but it takes some time. Plus I have to check which distro they support. I may try to get the latest ubuntu distro and upgrade the kernel to the most recent one. Second solution is to boot an emulated s390x Fedora 31 image using qemu. Could you please ping me in the libbepf-go slack channel, cannot find you?
I'm also searching for some bpf expert for s390x inside my team ;). I hope we can get some help!

[lmb]

If you want to work on this, I have the following questions / suggestions:

• How different are big and little endian eBPF? Is it just swapped registers? maybe the code gen is different? The library can do either of two things: load eBPF in any endianness, and then translate to host endianness at load time OR only load eBPF in host endian. The latter is definitely simpler, and might be the safe choice to start with (but would be harder to test without a VM.)
• The same goes for BTF (BPF Type Format)
• How can we skip the unsupported bpf-2-bpf test? It might be enough to amend https://github.com/cilium/ebpf/blob/master/internal/testutils/feature.go#L50 for EOPNOTSUP. Otherwise you might be able to write a new feature test: https://github.com/cilium/ebpf/blob/master/internal/feature.go#L38 The latter is might be a bit difficult, since feature tests currently don't take the host architecture into account.

For testing: as long as there is no big endian CI we might get away with adding some tests that compare some serialized eBPF from the asm package with known-good eBPF emitted by clang or something. You should also be able to amend the Makefile in testdata to compile the ebpf as big endian with -march=bpfeb.

[alicefr]

@lmb as @iii-i suggested, the bpf-2-bpf has been fixed from kernel 5.4 with the commit 1c8f9b91c456f5b47a377a0c8c5d4130fc39433a. I installed a newer kernel and it the link test was successful:

$  uname -r
5.7.0-rc1
$ go test -run TestLink
PASS
ok      github.com/cilium/ebpf  0.002s

[alicefr]

@lmb about enabling the CI for s390x. There are a couple of issues to do that.

The main problem how to get a kernel image for s390x. You're now fetching the kernel from https://github.com/cilium/ci-kernels. It is of course possible to cross-compile the kernel each time. But it takes ages. If you want we should add support for kernel cross-compilation in that repo

I had to do a small fix in virtme here: amluto/virtme#58 .

I tried to run an emulated guest and that required qemu-system-s390x 3.1. The default qemu-syste-s390x 2.7 didn't work for me.

If we need go inside the guest to run the test we need to generate correctly the rootfs for guest. Not sure if we might skip this by cross-compiling the test case for s390x and using s390x busybox binary .

This approach might be useful to test other arches but of course it requires some work and maintenance.