Deadlock caused by fork handlers

[loskutov]

I'm experiencing a deadlock; a fancified stacktrace is below. It looks like it's a bad idea to call library functions (which can call fork internally) having the fork-mutexes locked.

#0  0x00007feefe2196a0 in __GI___nanosleep
    at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x00007feefe24c7d4 in usleep (useconds=<optimized out>)
    at ../sysdeps/posix/usleep.c:32
#2  0x00007feefe716232 in _pkcs1h_threading_mutexLockAll ()
    at https://github.com/OpenSC/pkcs11-helper/blob/master/lib/pkcs11h-threading.c#L305
#3  0x00007feefe264578 in __run_fork_handlers
    (who=who@entry=atfork_run_prepare) at register-atfork.c:122
#4  0x00007feefe219707 in __libc_fork () at ../sysdeps/nptl/fork.c:58
#5  0x00007feefcf22712 in fork_exec_with_fds
    (intermediate_child=intermediate_child@entry=0, working_directory=working_directory@entry=0x0, argv=argv@entry=0x5650d153d1e0, envp=envp@entry=0x0, close_descriptors=close_descriptors@entry=1, search_path=search_path@entry=1, search_path_from_envp=0, stdout_to_null=0, stderr_to_null=0, child_inherits_stdin=0, file_and_argv_zero=0, cloexec_pipes=0, child_setup=0x0, user_data=0x0, child_pid=0x7ffc22321898, child_close_fds=0x7ffc223217b0, stdin_fd=-1, stdout_fd=5, stderr_fd=7, error=0x7ffc22321a58) at https://github.com/GNOME/glib/blob/master/glib/gspawn.c#L1647
#6  0x00007feefcf22fab in fork_exec_with_pipes
    (intermediate_child=intermediate_child@entry=0, working_directory=working_directory@entry=0x0, argv=0x5650d153d1e0, envp=envp@entry=0x0, close_descriptors=close_descriptors@entry=1, search_path=search_path@entry=1, search_path_from_envp=0, stdout_to_null=0, stderr_to_ro=0, cloexec_pipes=0, child_setup=0x0, user_data=0x0, child_pid=0x7ffc22321898, standard_input=0x0, standard_output=0x7ffc22321890, standard_error=0x7ffc22321894, error=0x7ffc22321a58) at https://github.com/GNOME/glib/blob/master/glib/gspawn.c#L1958
#7  0x00007feefcf233c6 in g_spawn_sync
    (working_directory=working_directory@entry=0x0, argv=<optimized out>, envp=envp@entry=0x0, flags=flags@entry=G_SPAWN_SEARCH_PATH, child_setup=child_setup@entry=0x0, user_data=user_data@entry=0x0, standard_output=0x7ffc223219e8, standard_error=0x7ffc223219f0, exit_status=0x7ffc223219e4, error=0x7ffc22321a58)
    at https://github.com/GNOME/glib/blob/master/glib/gspawn.c#L394
#8  0x00007feefcf23c07 in g_spawn_command_line_sync
    (command_line=command_line@entry=0x5650d153d100 "dbus-launch --autolaunch=790ef33180164b2a91d23c8c584cb1c2 --binary-syntax --close-stderr", standard_output=standard_output@entry=0x7ffc223219e8, standard_error=standard_error@entry=0x7ffc223219f0, exit_status=exit_status@entry=0x7ffc223219e4, error=error@entry=0x7ffc22321a58) at https://github.com/GNOME/glib/blob/master/glib/gspawn.c#L935
#9  0x00007feefd0b5cd0 in get_session_address_dbus_launch (error=error@entry=0x7ffc22321a58) at https://github.com/GNOME/glib/blob/master/gio/gdbusaddress.c#L1131
#10 0x00007feefd0b780a in get_session_address_platform_specific (error=0x7ffc22321a58) at https://github.com/GNOME/glib/blob/master/gio/gdbusaddress.c#L1559
#11 0x00007feefd0b780a in g_dbus_address_get_for_bus_sync (bus_type=bus_type@entry=G_BUS_TYPE_SESSION, cancellable=cancellable@entry=0x0, error=error@entry=0x0)
    at https://github.com/GNOME/glib/blob/master/gio/gdbusaddress.c#L1639
#12 0x00007feefd0c2d96 in get_uninitialized_connection (bus_type=bus_type@entry=G_BUS_TYPE_SESSION, cancellable=cancellable@entry=0x0, error=error@entry=0x0)
    at https://github.com/GNOME/glib/blob/master/gio/gdbusconnection.c#L7187
#13 0x00007feefd0c86ad in g_bus_get_sync (bus_type=bus_type@entry=G_BUS_TYPE_SESSION, cancellable=cancellable@entry=0x0, error=error@entry=0x0)
    at https://github.com/GNOME/glib/blob/master/gio/gdbusconnection.c#L7282
#14 0x00007feefd0a95fa in g_application_impl_register
    (application=application@entry=0x5650d1533890 [GApplication], appid=0x5650d1533790 "org.opensc.notify", flags=G_APPLICATION_NON_UNIQUE, exported_actions=0x5650d152ccd0, remote_actions=remote_actions@entry=0x5650d1533838, cancellable=cancellable@entry=0x0, error=0x0) at https://github.com/GNOME/glib/blob/master/gio/gapplicationimpl-dbus.c#L545
#15 0x00007feefd0a6464 in g_application_register (application=0x5650d1533890 [GApplication], cancellable=0x0, error=0x0) at https://github.com/GNOME/glib/blob/master/gio/gapplication.c#L2120
#16 0x00007feefd7a21e5 in C_Initialize () at /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
#17 0x00007feefe71f6e9 in pkcs11h_addProvider
    (reference=0x5650d1515008 "/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so", provider_location=<optimized out>, allow_protected_auth=0, mask_private_mode=0, slot_event_method=0, slot_poll_interval=0, cert_is_private=0) at https://github.com/OpenSC/pkcs11-helper/blob/master/lib/pkcs11h-core.c#L778

[alonbl]

This is opensc provider which forks during C_Initialize?!?! In what case does it happen? PKCS#11 provider must not fork I can think of so many things that may go wrong in this case.

[loskutov]

Well, I can only tell it happens when I'm trying to use OpenVPN with a Yubikey token. If I rebuild pkcs11-helper with threading support disabled, it appears to work.

[alonbl]

On Mon, Oct 8, 2018 at 8:06 PM Ignat Loskutov ***@***.***> wrote: Well, I can only tell it happens when I'm trying to use OpenVPN with a Yubikey token. If I rebuild pkcs11-helper with threading support disabled, it appears to work.

How is Yubikey related to OpenSC? I see something with gnome in stack that is triggered by opensc provider. See #5, I added support for unsafe fork mode, not sure if openvpn developers used this already or not.

[loskutov]

Yubikey has a PIV applet, which is accessed via PKCS#11, as far as I understand.

[loskutov]

It looks like the issue is caused by the notifications UI: https://github.com/OpenSC/OpenSC/blob/master/src/ui/notify.c#L446
https://github.com/OpenSC/OpenSC/blob/master/src/pkcs11/pkcs11-global.c#L241

[loskutov]

So, if forks are forbidden inside C_Initialize, does it mean it's an OpenSC issue or OpenVPN should be aware of fork-unsafe providers and use the fork-unsafe mode of pkcs11-helper (which it currently doesn't)?

[alonbl]

PKCS#11 spec requires to perform C_Initialize for all loaded providers post fork(), and C_Finalize if providers are not being used. The information of what providers are loaded by the process is not something a single provider has access to. Performing fork() also inherit resources to child process, and in some cases may allow child process to access cryptographic resources. A PKCS#11 provider that performs fork() violates the PKCS#11 spec and may cause lockout of parent process.

I am unsure how to handle this within the constraint of the standard and common sense, in any case please try to patch openvpn to use unsafe fork #5 and hope for the best...

One alternative to fix this is to have a separate program of pinentry that listens to unix domain socket within user's home directory (or /run/user/$uid) and for the PKCS#11 provider to connect into that program in order to query for credentials.

[alonbl]

@frankmorgner : see this one as well.

[frankmorgner]

Violating PKCS#11 by forking is debatable. After all, its pkcs11-helper that deadlocks itself, because it doesn't expect a fork inside the atfork handler. I've created #17 to work around this limitation and to hopefully fix this issue (currently untested).

[rosly]

Would be possible to fix this issue? It blocks us from using Yubikey with OpenVPN.

[alonbl]

On Tue, Feb 26, 2019 at 7:05 PM Radosław Biernacki ***@***.***> wrote: Would be possible to fix this issue? It blocks us from using Yubikey with OpenVPN.

Does yubikey also forks in C_Initialize?

[loskutov]

My usecase is also using OpenVPN with Yubikey, which is worked around by building pkcs11-helper without threading support.