Macro overrides for customizable Lua compile-time options

[sagebind]

(Please forgive me for the wall of text. It is an interesting read, I promise.)

I am currently investigating into using this crate in a multithreaded manner. In order to do so, I need to override the Lua definitions for the lua_lock() and lua_unlock() macros, which brought up the larger question of how these macros can be overridden. Since Lua is meant to be customizable for many different uses, I think this is an important discussion to have and something we should try to solve.

The main issue as to why this is not easy is because Lua is compiled before this crate, and not with it. C and Rust can't be mixed, so this isn't something we can solve to my knowledge.

I see three options currently, though I'd love to see if anyone else has better ideas than these admittedly poor options:

1. Add the ability to include custom C code during the compilation process by modifying Lua's Makefile (which we can now do, thanks to Bundle Lua rather than download it #59). This would be a feature added to the build script.
2. Don't allow consumers of the crate to override macros, and use a combination of C and Rust to pick-and-choose how we want to handle specific macros. For example, write hardcoded implementations of lua_lock() and lua_unlock() and make thread safety a non-optional feature.
3. Use function pointers and stubbing to override definitions at runtime. This would have a small runtime overhead.

Personally, (1) does not sound desirable at all, even if it can be done. I don't feel like writing any C code in a Rust application that depends on lua.

(2) is not desirable either, especially for the locking case. Locking incurs a notable runtime overhead that we do not want to force on users in a single-threaded application. Since this is compile-time, it might be near-impossible to be able to turn thread-safety off.

(3) isn't pretty, but it kind of works and isn't as much of a runtime cost as (2) could be for some users. Calling all overridable functions would have an additional branch instruction checking for NULL before making a jump to an implementation.

Here's some of the ugliness of (3) so that you can assess if it is worth it...

Adding externs for values that are overridable in lua-source/overrides.h (we must choose what overrides we want to support):

typedef struct lua_State lua_State;

#define lua_lock(L)     if (LUA_LOCK_STUB != 0) { (*LUA_LOCK_STUB)(L); } else
#define lua_unlock(L)   if (LUA_UNLOCK_STUB != 0) { (*LUA_UNLOCK_STUB)(L); } else

extern void (*LUA_LOCK_STUB)(lua_State* L);
extern void (*LUA_UNLOCK_STUB)(lua_State* L);

Define references for the above symbols initialized to null (lua-source/overrides.c):

void (*LUA_LOCK_STUB)(lua_State* L) = 0;
void (*LUA_UNLOCK_STUB)(lua_State* L) = 0;

Back to Rust, we define a macro that consumers of the crate can use to set these overridable function pointers:

macro_rules! lua_override {
    ($SYMBOL:ident, $f:path) => {
        {
            extern "C" {
                static mut $SYMBOL: *mut $crate::libc::c_void;
            }
            unsafe {
                $SYMBOL = {
                    extern "C" fn fn_stub(state: *mut $crate::ffi::lua_State) {
                        $f(state);
                    }
                    fn_stub
                } as *mut $crate::libc::c_void;
            }
        }
    };
}

This indeed works! A usage example:

#[macro_use]
extern crate lua;

fn main() {
    lua_override!(LUA_LOCK_STUB, lock);
    lua_override!(LUA_UNLOCK_STUB, unlock);

    let mut state = lua::State::new();
    state.do_string("print('hi')");
}

fn lock(state: *mut lua_State) {
    println!("lua_lock called");
}

fn unlock(state: *mut lua_State) {
    println!("lua_unlock called");
}

This prints "hi" between a whole lot of "lua_lock called" and "lua_unlock called". If this approach is desirable, as it looks decent to the consumer, then I can open a PR with a tidier implementation.

Are any of these good ideas? How do we want to handle macro overrides?

[SpaceManiac]

I agree, option 1 is right out; 2 is possible, but reduces flexibility for other consumers of the lua-sys crate (should there ever be any). 3 is the best option of those you present; a little more switching but maximum flexibility. I was thinking about using weak symbols to keep the override resolution to compile time (no-op functions as weak symbols, overridable by the consumer), but it might get less portable and also reduces flexibility.

We'll have to look at which Lua macros it makes sense to allow overriding - and whether we want to allow configuration of, say, the Integer types, if that's even possible (certainly it's much trickier than function overrides).

[sagebind]

Weak symbols are an ELF-specific thing IIRC, which limits the platforms supported significantly. Notably, I believe clang does not support weak symbols anyway.

[SpaceManiac]

Mm, seems you're right; weak symbols would be way trickier. I think your function stubbing is probably the best route, then. That macro looks a little unwieldy and can probably be simplified, but its invocation seems great. There's a variety of macros that could be overridden in llimits.h, but the most important ones are probably lua_lock, lua_unlock, luai_threadyield, and the luai_userstateX macros.