Perl spits out warning about locales

[MikeFultonDev]

The texinfo test suite has a number of failures due to a message about locales.
See: zopencommunity/texinfoport#17 for details.

@IgorTodorovskiIBM not sure if this is related to recent changes in Perl or not - perhaps an env var isn't set right now - but this
could always have been there (new tests now running).

[MikeFultonDev]

Digging in a little bit, it appears to be because 'ispunct' is giving the wrong answers for the UTF8 locale. Here's a testcase that shows when the test is built in 'ASCII-mode' ispunct gets the wrong answer for the UTF-8 locale and when built in 'EBCDIC-mode' the setlocale to the UTF8 locale fails.

test>clang -fzos-le-char-mode=ascii punctest.c

test>./a.out

C: ispunct('#'):256
UTF-8: ispunct('#'):0

test>clang punctest.c
test>./a.out

C: ispunct('#'):16
Unable to set locale to UTF-8

test>cat punctest.c

#include <ctype.h>
#include <stdio.h>
#include <locale.h>

int main() {
  printf("C: ispunct('#'):%d\n", ispunct((unsigned char) ('#')));
  if (!setlocale(LC_ALL, "en_US.UTF-8")) {
    fprintf(stderr, "Unable to set locale to UTF-8\n");
    return 8;
  }
  printf("UTF-8: ispunct('#'):%d\n", ispunct((unsigned char) ('#')));
  return 0;
}

[MikeFultonDev]

Defining your own locale: https://www.ibm.com/docs/en/zos/3.1.0?topic=lu-under-zos-shell and https://www.ibm.com/docs/en/zos/3.1.0?topic=descriptions-localedef-define-locale-environment and https://www.ibm.com/docs/en/zos/3.1.0?topic=lsf-lc-ctype-category

[MikeFultonDev]

Creating a custom locale: https://www.ibm.com/docs/en/zos/3.1.0?topic=ilcs-customizing-locale

[khwilliamson]

Every OS that I've tested with perl has had locale-related bugs. Darwin is the worst. Therefore, we have developed workarounds for a bunch of things.

Though we wouldn't be able to work around not being able to have a UTF-8 locale on EBCDIC. I imagine this is because support for UTF-EBCDIC is basically gone. Perl uses it because it means there's very little EBCDIC-specific code in our source. Almost all but a few very low level macros don't need to know if what is underlying is UTF-8 or UTF-EBCDIC. So the code works regardless without needing special handling.

But we do have code that is specific to various platforms that could take care of ispunct('#') returning the wrong thing. Is it just this character, or most. Maybe that area of the OS hasn't been updated to consider ASCII mode. If so, that would be a ticket against the OS. But perl could easily add a layer to return the correct values just for z/OS.

z/OS locale code is basically unusable with threads. The design is that after the first non-main thread is started, all setlocale calls fail. That means it is thread-safe, but useless, except to set up a locale at the beginning of the program and use that for the rest. A web server that takes requests in many languages could not be written as a single threaded process for z/OS. Perl can often change the LC_NUMERIC locale category to accommodate locales where the radix character is a comma (most of Europe) but the software assumes it is a dot.

So I got somewhat excited when I noticed that our probes for 2008 thread-safe locale libc functions, such as newlocale now show that there are entry points for them in z/OS libc. But they aren't documented, and are lacking essential components. Perhaps this is evidence of a future direction.

By the way, I wonder if the punctest.c program actually needs to cast the input to ispunct to unsigned char I know that early Unix libc's, didn't care if the input character was signed or not. They did this by having an extended table that could accommodate either.