[Proposal] Windows: locale issues

[cyanidle]

Considering that mate.c is a script, and will probably deal with CLI args (#11) some kind of locale support is required?

What if filenames are not in ASCI?

main() received argc, argv in current Code Page on windows, which may be a lossy conversion. CLI should be parsed with UTF16 (or convert it to UTF8) with GetCommandLine() or smth like that.

Maybe just doing everything in UTF8 on windows is fine. But to avoid problems with CLI it may be useful to hide argc + argv from user completely with something like this

// hides StartBuild(), ParseCLI(), EndBuild()
MATE_SCRIPT() { 
   CreateExecutable()
   AddFile()
   InstallExecutable()
}

#define MATE_SCRIPT() \
  void mate_main(void); \
  int main(...) { __StartBuild(argc, argv); mate_main(); __EndBuild(); } \
  void mate_main(void)

[TomasBorquez]

I feel like it's too much magic at that point, being a bit more verbose and just having to give argc and argv manually to StartBuild is fine to me.

But I do agree locale support might be an issue, though for the first release I want to keep it simple, this is more of an edgecase though if you wanna work on it I'm all for it :P

We would need some test cases for cyrillic, chinese and japanese for now, and we can add it to CI

[sten-code]

A little off-topic: I like the idea of having a kind of MATE_SCRIPT macro, but this implementation makes it look like its possible to have multiple scripts.

Another option would be a kind of MATE_TARGET which would allow you to define individual targets, and then hide the main function inside the mate.h header file.

#define MATE_IMPLEMENTATION
#include "mate.h"

MATE_TARGET(main) {
    // build rules
}

MATE_TARGET(examples) {
    AddDependencies("main");
    // build rules
}

Although I like this API, I'm not entirely sure this would be a good idea, since it hides a lot of functionality.

[cyanidle]

@sten-code Now lets add conditionally built target based on OS for example. This breaks down immediately sadly, so I am with Tomas here, less magic the better.

[sten-code]

@cyanidle Yea I know, it was just an idea based on what you were suggesting.

I'm pretty sure it would be possible to make this work with conditional builds based on the target OS, but in the long term, this wouldn't scale well. It also doesn't give enough freedom to developers, and basically turns this into just another build system.

[AverageUser125]

I think I got some idea on how to solve the window locale issue atleast.

void ParseArgs(int* argc, char*** argv);

which can be called at the start of main like this:

int main(int argc, char** argv) {
    StartBuild();
    ParseArgs(&argc, &argv);
    // now argv contains UTF-8 arguments on all platforms
}

On Windows, ParseArgs will get the full command line in UTF-16 using GetCommandLineW(). Then it will parse it into arguments using CommandLineToArgvW (such as the implementation from WINE here) instead of the shell32 cause requiring linking that on windows will cause more problems. Then each UTF-16 argument is then converted to UTF-8 using WideCharToMultiByte.

• The temporary UTF-16 argument buffers returned by the custom CommandLineToArgvW are only needed briefly, so they can be allocated with malloc and freed immediately after conversion or just also thrown on the arena. The final UTF-8 strings can be stored in the arena or similar.
• Creating a function the uses the UTF-16 command line and parses and converts it to UTF-8 on the fly is another option, and even if we go this route atleast it is possible to take inspiration from the WINE implementation.
• I pass argc and argv by pointer so ParseArgs can replace them in-place. This is to ensure the rest of the program sees the normalized UTF-8 arguments and it should prevent future problems.
• Also I am pretty sure that the final argc will never be diffrent from the one that main gives you, but just to be sure it will be passed by-pointer as well, maybe an assert to make sure of this should be added.
• Maybe StartBuild will take the arguments instead of ParseArgs, but arguments are not always used so I am not sure.
• On Linux this function should just use the arguments as they are and not change them.
• One extra idea is to store the size of each argument immediately before the argument string, so you could retrieve it later by doing something like:

uint32_t size = *((uint32_t*)&argv[0][-4]);