OpenSCManagerW crashes in ARM64EC when called from x86_64

[Michael-Ost-BoseProfessional]

System Details

Platform: Ubuntu Linux 25.10 (UTM VM)
Hardware: MacBookPro M4
Application: x86_64 LDSvc.exe (PACE License Support system service)
Hangover: fork from Hangover 11.4 (https://github.com/Michael-Ost-BoseProfessional/hangover). Emulating with FEX.

Problem Description

When x86_64 code emulated by Hangover FEX calls OpenSCManagerW in ARM64EC it crashes. The problem is that the FEX code doesn't set up argument registers correctly for a variadic function like OpenSCManagerW. I've explored two solutions:

1. Fix the assembler in hangover/wine/dlls/rpcrt4/ndr-stubless.c NdrClientCall2 (which is a proxy to OpenSCManagerW) to fix up the x4 register to point to variadic argument 3.
2. Fix up the ExitFunctionEC function in hangover/fex/Source/Windows/ARM64EC/Module.S to leave x4 in the right state.

I explored #1 because I already had a fork of Wine, but didn't have a fork of FEX to play with, so it was easier to fix Wine. And the fix is a little scary and hard to test. So I liked that this fix is more localized to the specific crash I was seeing. Here's a commit showing what I did Michael-Ost-BoseProfessional/wine@cc6dbd8.

But as I look closer at it, #2 might be the better, cleaner fix. I am open to making this change to FEX to see if it also fixes my crash.

What's your preference?

[AndreRH]

Hi, I'd prefer if option 2 goes upstream into FEX.
Though it looks a bit like something they should have noticed, would be a good first thing to connect with them via discord or a issue to check.

[AndreRH]

Any update on this?

[Michael-Ost-BoseProfessional]

Hi Andre,

This was a deep one that I eventually fixed, but I'm not sure I got to the bottom of it.

I did a lot more exploration of this issue and realized I wasn't quite on track with the above description. The problem seems more related to -something- that happens to stack frames when DLLs get loaded. I didn't find the smoking gun, but I did write a simple test that demonstrates the crash and a patch to NdrClientCall2 that fixes it. The problem is in Wine and not FEX.

For background, the ARM64EC spec for variadic arguments is that x2=arg1 x3=arg2, x4 points to a buffer of overflow args, and x5 points to the size of the buffer at x4.

OpenSCManagerW calls svcctl_OpenSCManagerW (IDL), which sets this arrangement up correctly by putting the overflow args (dwAccesMask, handle) on the local stack at x4. NdrClientCall2 is called. It is supposed to set up "stack_top" to pass to NrdpClientCall2 as a contiguous buffer with the first two args x2/x3 just before the overflow args at x4. The assembler does by inserting x2 and x3 in the 16 bytes before where x4 points. It does x4 -= 16 assuming and passes that as "stack_top", assuming that makes a contiguous buffer.

But... the first time you call OpenSCManagerW this causes DLLs to load (rpcrt4.dll, if memory serves). When this happens x4 is pointing -indirectly- to the overflow args; pointing at a descriptor {args, len} instead of pointing directly to the args in x4/x5. This is problem 1.

Problem 2 is that the trick to insert x2/x3 before x4 can't work, since x4 is no longer pointing the the stack frame. Problem 3 is that the indirect pointer of x4 is passed to NdrpClientCall2 as 'stack_top'. The invalid values in that buffer lead to a crash (and a hard to find and diagnose crash at that!).

What I noticed was that when that DLLs first loads, there is extra space is inserted on the stack. My solution -- if I detect that gap, I build up a local array of values to pass in as 'stack_top'. If there is no gap, the original code is employed.

This is arguably more of a workaround than a real fix. I -think- problem #1 is a deeper real bug in how Wine handles variadic arguments in ARM64EC when loading a DLL. But I ran out of time before finding where that actually happens. I'm going to live with my fix, but I could see that you might not want it.

I was also unsure about how to integrate my test. Should it go into wine proper?

Please take a look and advise.

/M

0001-Fix-Crash-in-first-call-of-OpenSCManagerW-NdrClientC.patch

test.gz