arm libc has slow memset/memcpy implementations (and probably other things?)

[faithanalog]

I'm working on a rust GBA project, and I'm using picolibc (it is picolibc, right?) from wf to provide some libc stuff to some other libraries im linking in.

Note: I am not using the one in arm-none-eabi/gba/lib/, because that one didn't have malloc. I'm using the one in arm-none-eabi/lib/arm7tdmi/. But they both have this problem, as does the arm9 libc.a

I notice that, for memory initialization/moving, my rust code was using the memset/memcpy implementations from picolibc instead of the default provided by llvm. this is a decision that gets made at link time, because at compile time they are just emitted in the assembly as refs to the aeabi memset/memcpy. I don't fully understand how the decision works, I just know that's how the decision is happening in my project (and for a project without picolibc it wouldnt do that). And I think if I provided my own aeabi mem operations I wouldn't have noticed this, though it'd probably be slowing down my C libraries that use the libc mem functions.

ANYWAY, that's just context, it's not actually a problem.

The Problem

The problem is that picolibc's memcpy and memset are doing bytewise copies/sets only!!! Like ldrb/strb loops. I noticed this because the initializer for a 300-word array (in iwram) was taking something like 30,000 cycles. which seemed excessive to me. So I replaced it with a DMA-memset and got the 600 cycles you'd expect from that, but I was still confused why the default initializer was slow.

Long story short, picolibc has some uh... surprising, to me? behavior when you compile with -Os.

void *
__inhibit_loop_to_libcall
memcpy (void *__restrict dst0,
	const void *__restrict src0,
	size_t len0)
{
  char *dst = dst0;
  const char *src = src0;
#if !(defined(PREFER_SIZE_OVER_SPEED) || defined(__OPTIMIZE_SIZE__))
  /* snipped for brevity: basically all the things that make a memcpy
     fast for the word or multi-word aligned part of a memory block
     by transferring 4-byte aligned chunks as 32-bit words, unrolling to
     copy 16-bytes per loop iteration for big chunks etc. */
#endif

  while (len0--)
    *dst++ = *src++;

  return dst0;
}

The defined(PREFER_SIZE_OVER_SPEED) makes some sense, but the check for defined(__OPTIMIZE_SIZE__) is a compiler builtin that gets set if you build with -Os (src gcc docs]. So basically if you use -Os you get extremely slow memory operations.

I don't know if thats what the wf picolibc is built with, but given it's normally totally fine and reasonable to be using -Os to build C code, I figure that's what it is.

You can see this in the disassembly:

> arm-none-eabi-objdump --disassemble='memcpy' /opt/wonderful/toolchain/gcc-arm-none-eabi/arm-none-eabi/lib/arm7tdmi/libc.a
 | grep -A20 '.text.memcpy:'
Disassembly of section .text.memcpy:

00000000 <memcpy>:
   0:	e2403001 	sub	r3, r0, #1
   4:	e0812002 	add	r2, r1, r2
   8:	e1510002 	cmp	r1, r2
   c:	012fff1e 	bxeq	lr
  10:	e4d1c001 	ldrb	ip, [r1], #1
  14:	e5e3c001 	strb	ip, [r3, #1]!
  18:	eafffffa 	b	8 <memcpy+0x8>

I ran this command on the gba libc, generic arm7tdmi libc, arm7 thumb libc, arm9 libc, and they've all got roughly this disassembly. Which makes sense, I'd kinda find it weird if they didnt given I assume they're all built in about the same way, I just wanted to check to be able to say I did.

memset has the same thing going on.

Affected functions

I haven't read the code for all of these but just a guess by grep. its mostly mem/str stuff.

> ag -l '__OPTIMIZE_SIZE__'
newlib/libc/posix/dirname.c
newlib/libc/string/strstr.c
newlib/libc/string/strcat.c
newlib/libc/string/memrchr.c
newlib/libc/string/strncmp.c
newlib/libc/string/stpcpy.c
newlib/libc/string/strcasestr.c
newlib/libc/string/memcmp.c
newlib/libc/string/strncat.c
newlib/libc/string/memchr.c
newlib/libc/string/mempcpy.c
newlib/libc/string/strcmp.c
newlib/libc/string/strchr.c
newlib/libc/string/rawmemchr.c
newlib/libc/string/memccpy.c
newlib/libc/string/strncpy.c
newlib/libc/string/memmem.c
newlib/libc/string/memcpy.c
newlib/libc/string/strlen.c
newlib/libc/string/memmove.c
newlib/libc/string/strcpy.c
newlib/libc/string/stpncpy.c
newlib/libc/string/memset.c
newlib/libc/include/ctype.h
newlib/libc/stdlib/gdtoa-gethex.c
newlib/libc/stdlib/a64l.c
newlib/libc/stdlib/mprec.h
newlib/libc/tinystdio/atof_engine.c
newlib/libc/tinystdio/atod_engine.c
newlib/libc/iconv/ccs/ccs.h
newlib/libc/stdio/fread.c
newlib/libc/stdio/findfp.c
newlib/libc/stdio/vfwprintf.c

Some extra context

Here you can see one of my rust functions, eventually using it in the final binary

Before linking, assembly has a __aeabi_memcpy4

> cargo asm --bin gba-template gba_template::ui::ScrInstrEdit::init_instrs 0 2>/dev/null | grep memcpy
	bl __aeabi_memcpy4
	bl __aeabi_memcpy

After linking, both calls are to normal memcpy, which again, normally not the end of the world.

> arm-none-eabi-objdump --disassemble='_ZN12gba_template2ui12ScrInstrEdit11init_instrs17h7c68fef94f8634aeE' target/armv4t-n
one-eabi/release/gba-template | grep mem
 8006b08:	eb02c055 	bl	80b6c64 <memcpy>
 8006b2c:	eb02c04c 	bl	80b6c64 <memcpy>
vi@localhost ~/p/gbatrack (main)> 

and the offending implementations:

vi@localhost ~/p/gbatrack (main) [1]> arm-none-eabi-objdump --disassemble=memset target/armv4t-none-eabi/release/gba-template

target/armv4t-none-eabi/release/gba-template:     file format elf32-littlearm


Disassembly of section .text:

080b6e14 <memset>:
 80b6e14:	e1a03000 	mov	r3, r0
 80b6e18:	e0802002 	add	r2, r0, r2
 80b6e1c:	e1530002 	cmp	r3, r2
 80b6e20:	012fff1e 	bxeq	lr
 80b6e24:	e4c31001 	strb	r1, [r3], #1
 80b6e28:	eafffffb 	b	80b6e1c <memset+0x8>

Disassembly of section .data:
vi@localhost ~/p/gbatrack (main)> arm-none-eabi-objdump --disassemble=memcpy target/armv4t-none-eabi/release/gba-template

target/armv4t-none-eabi/release/gba-template:     file format elf32-littlearm


Disassembly of section .text:

080b6db0 <memcpy>:
 80b6db0:	e2403001 	sub	r3, r0, #1
 80b6db4:	e0812002 	add	r2, r1, r2
 80b6db8:	e1510002 	cmp	r1, r2
 80b6dbc:	012fff1e 	bxeq	lr
 80b6dc0:	e4d1c001 	ldrb	ip, [r1], #1
 80b6dc4:	e5e3c001 	strb	ip, [r3, #1]!
 80b6dc8:	eafffffa 	b	80b6db8 <memcpy+0x8>

Disassembly of section .data:

[asiekierka]

Yes, that's correct! What follows is my attempt to clarify what's going on:

• The GBA-specific build of picolibc is deprecated; I wanted to avoid using thread local storage on platforms which do not need it, but that leads to ABI incompatibilities with libraries built on top of libc, such as libstdc++. I didn't get around to removing the package. yet
• I build picolibc with -Os because the platforms I target thus far (GBA, NDS via BlocksDS) have low RAM, and not in all cases the benefit of being able to store code in ROM.
• The slow speed of memcpy in picolibc itself was not considered a big issue, as both BlocksDS and my own GBA library bundle agbabi, which includes, among other things, fast implementations of memcpy, memmove and memset.
• The GBA target in general is neglected, so the GBA library did not end up being recommended; I released it early specifically because GValiente wanted a "second-source" binary toolchain for the Butano project. As I'm currently somewhat burned out on retro development, and have no particular GBA-specific projects I plan to work on, I have no plans on working on it further at this time.

The best solution for users not making direct use of Wonderful's targets (for reasons as described above) is most likely to patch our fork of picolibc to provide agbabi's implementations by default, or - at minimum - use the speed-optimized memcpy() variant. However, I have no ETA for that.

[faithanalog]

no problem for me! I had added "build my own picolibc" to my list of things to do eventually, when I want some free performance. Though, if I can figure out the linker decides which library to pull a function from, maybe I can provide my own implementations like how you're doing with agbabi. I need to version bump my dependency of the rust gba crate anyway to see what happens when I do that; they had their own mem functions for awhile, and then took them out for a bit, and then put them back in again, and I added the dependency to my project in that window when they weren't there.

And good to know about the gba-specific builds. I was worried I was doing something wrong by not using them, good to hear i'm actually doing something right.