FFI rework 2

[sydhds]

Introduction

As Waku requested to pass data (aka bytes) in Big endian format, we start to think about how we pass arguments to the FFI (Foreign Function Interface) of Zerokit.

Current status

Using ffi in Zerokit is done by passing bytes as arguments to the FFI interface of Zerokit. For instance, here is the ffi generate_rln_proof_with_witness function:

pub extern "C" fn generate_rln_proof_with_witness(
    ctx: *mut RLN,
    input_buffer: *const Buffer,
    output_buffer: *mut Buffer,
) -> bool {
    call_with_output_arg!(
        ctx,
        generate_rln_proof_with_witness,
        output_buffer,
        input_buffer
    )
}

which is calling Rust function:

    pub fn generate_rln_proof_with_witness<R: Read, W: Write>(
        &mut self,
        mut input_data: R,
        mut output_data: W,
    ) -> Result<(), RLNError> {
        let mut serialized_witness: Vec<u8> = Vec::new();
        input_data.read_to_end(&mut serialized_witness)?;
        let (rln_witness, _) = deserialize_witness(&serialized_witness)?; // deserialize from input_data
        let proof_values = proof_values_from_witness(&rln_witness)?;

        let proof = generate_proof(&self.proving_key, &rln_witness, &self.graph_data)?;

        // Note: we export a serialization of ark-groth16::Proof not semaphore::Proof
        // This proof is compressed, i.e. 128 bytes long
        proof.serialize_compressed(&mut output_data)?; // serialize into output_data
        output_data.write_all(&serialize_proof_values(&proof_values))?;
        Ok(())
    }

So for the developer this requires:
1- Allocate two buffers (input & output)
2- Serialize the arguments into the input buffer
3- Call the Rust function which deserialize the input buffer then serialize into the output buffer

Problems

• Require to allocate data for serialized data
• Require to deserialize input(s) then call then serialize output -> Overhead - decrease perf
• Require to know how to serialize the data - any changes in the serialized data or arguments will not be seen (just bytes are passed to functions)
• Require complex documentation (of the expected serialized data)
• Require different functions if someone decides to pass bytes in BigEndian (instead of LittleEndian)
• Dev maintenance of a lot of functions related bytes <-> Struct / Enum translation

FFI functions list (requiring some serialization / deserialization)

Merkle tree api (Not exhaustive list):

function	argument 1	Rust type
set_leaf	input_buffer	Fr
get_leaf	output_buffer	Fr
init_tree_with_leaves	input_data	Vec<Fr>

Zk api:

function	argument 1	Rust type	argument 2	Rust type
prove	input_buffer	RLNWitnessInput	output_data	Proof<Curve>
verify	input_buffer	Proof<Curve> + RLNProofValues
generate_rln_proof	input_buffer	RLNWitnessInput + RLNProofValues	output_buffer	Proof<Curve>
verify_rln_proof	proof_buffer	Proof<curve> + RLNProofValues + signal	proof_is_valid_ptr	bool
verify_with_roots	proof_buffer	Proof<curve> + RLNProofValues + signal	roots_buffer	Vec<Fr>

Utils api:

function	argument 1	Rust type	argument 2	Rust type	argument 3	Rust type
key_gen	output_data	(Fr, Fr)
recover_id_secret	input_proof_data_1	(.., RLNProofValues)	input_proof_data_2	(.., RLNProofValues)	output_data	Fr

Solution

Solution for C: Opaque struct managed by Rust

A common way to pass data in FFI interface is to use opaque struct. The idea is to provide object (allocated by Zerokit rust lib) + a set of defined functions to manipulate them in C.

Note:

• Provided examples use the safer_ffi crate as it allows to write FFI without unsafe blocks.
  • There are multiple other advantages of using safer_ffi (see this & this) but this will not be discussed here.
  • safer_ffi has all the tooling to generate C headers automatically
• All C examples were tested with valgrind to check for memory leaks

As a basic example, we could define:

use ark_bn254::Fr;
use ark_std::{rand::thread_rng, UniformRand, Zero};
use safer_ffi::{derive_ReprC, ffi_export};
use safer_ffi::boxed::Box_;
use safer_ffi::prelude::{repr_c};

#[derive_ReprC]
#[repr(opaque)]
#[derive(Debug)]
struct CFr(Fr);

impl Default for CFr {
    fn default() -> Self {
        Self(Fr::zero())
    }
}

#[ffi_export]
fn cfr_debug(cfr: Option<&CFr>) {
    println!("{:?}", cfr);
}

// Rust original function (in protocol.rs)
fn keygen_() -> (Fr, Fr) {
    let mut rng = thread_rng();
    let identity_secret_hash = Fr::rand(&mut rng);
    // Test only
    // let id_commitment = poseidon_hash(&[identity_secret_hash]);
    let id_commitment = Fr::rand(&mut rng);

    // println!("keygen: {} {}", identity_secret_hash, id_commitment);
    (identity_secret_hash, id_commitment)
}

#[ffi_export]
fn keygen() -> repr_c::Vec<CFr> {
    let (identity_secret_hash, id_commitment) = keygen_();
    vec![CFr(identity_secret_hash), CFr(id_commitment)].into()
}

#[ffi_export]
fn zero<'a>() -> repr_c::Box<CFr> {
    // Init a CFr to 0 value
    Box_::new(CFr::default())
}

#[ffi_export]
fn cfr_free(cfr: repr_c::Box<CFr>) {
    drop(cfr);
}

// Vec<CFr> functions

#[ffi_export]
fn vec_cfr_get(v: Option<&repr_c::Vec<CFr>>, i: usize) -> Option<&CFr> {
    // Get element i of given vector of CFr
    v.and_then(|v| v.get(i))
}

#[ffi_export]
fn vec_cfr_free(v: repr_c::Vec<CFr>) {
    drop(v);
}

And thus this lib can be used in C:

#include <stdlib.h>
#include <stdio.h>

#include "witness.h"

int main (int argc, char const * const argv[])
{
    CFr_t* z0 = zero();
    cfr_debug(z0);
    cfr_free(z0);

    // Test we can use cfr_debug on NULL values
    cfr_debug(NULL);

    // Use keygen function
    Vec_CFr_t generated = keygen();

    CFr_t* id_secret_hash = generated.ptr;
    cfr_debug(id_secret_hash);
    CFr_t* id_co = vec_cfr_get(&generated, 1);
    cfr_debug(id_co);
    // Test vec_cfr_get with wrong values
    CFr_t* id_test = vec_cfr_get(&generated, 2);
    cfr_debug(id_test);

    vec_cfr_free(generated);

    return EXIT_SUCCESS;
}

So at the end, we would define opaque struct for:

Rust Type	Rust ffi Type
Fr	CFr
Vec<Fr>	repr_c::Vec<CFr>

And for each type, the following functions:

• Initialize functions: CFr::Zero, CFr::Max, CFr::from_u8, ..., CFr::from_u64
• Serialization & Deserialization functions: CFr::serialize_le, CFr::deserialize_le, ...

Based on CFr struct, we could declare non opaque struct like:

#[derive_ReprC]
#[repr(C)]
#[derive(Debug)]
pub struct CRLNProofValues {
    y: repr_c::Box<CFr>,
    nullifier: repr_c::Box<CFr>,
    root: repr_c::Box<CFr>,
    x: repr_c::Box<CFr>,
    external_nullifier: repr_c::Box<CFr>,
}

#[derive_ReprC]
#[repr(C)]
#[derive(Debug)]
pub struct CRLNWitnessInput {
    identity_secret: repr_c::Box<CFr>,
    // user_message_limit: repr_c::Box<CFr>,
    // message_id: repr_c::Box<CFr>,
    path_elements: repr_c::Vec<CFr>,
    identity_path_index: repr_c::Box<[u8]>,
    // x: repr_c::Box<CFr>,
    // external_nullifier: repr_c::Box<CFr>,
}

And use them in C:

int main(void) {
    
    {
        CFr_t* y = cfr_one();
        CRLNProofValues_t* proof_v = proof_values_from(&y, ...);
        cfr_debug(proof_v->y);
        proof_values_free(proof_v);
    }

    {
        Vec_CFr_t id_generated = keygen();
        CFr_t* id_secret_hash = vec_cfr_get(&id_generated, 0);
        Vec_CFr_t path_elem = get_path_elements();
        slice_ref_uint8_t path_index = get_path_index_2();
        // for(int i=0; i<path_index.len; i++) {
        //     printf("[%d] v: {%d}\n", i, path_index.ptr[i]);
        // }

        CRLNWitnessInput_t* witness_1 = rln_witness_input_new(id_secret_hash, &path_elem, path_index);
        rln_witness_input_debug(witness_1);

        vec_cfr_free(generated);
        vec_cfr_free(path_elem);
        rln_witness_input_free(witness_1);
    }
}

Solution for Wasm

Current rln-wasm also use the serialization & deserialization way to pass arguments ffi (+ use the wasm-bindgen crate to generate the JS / TS bindings).

But wasm-bindgen allow the use of opaque structure and so we could define something like:

use ark_bn254::Fr;
use js_sys::BigInt;
use wasm_bindgen::prelude::wasm_bindgen;

#[wasm_bindgen]
#[derive(Clone, Copy, Debug)]
pub struct WasmFr(Fr);

#[wasm_bindgen]
impl WasmFr {

    pub fn zero() -> Self {
        Self(Fr::from(0))
    }

    pub fn new(value: u64) -> Self {
        Self(Fr::from(value))
    }

    pub fn from_bigint(value: BigInt) -> Self {
        Self::from(value)
    }

    pub fn debug(&self) -> String {
        format!("{:?}", self)
    }
}

impl From<BigInt> for WasmFr {
    fn from(value: BigInt) -> Self {
        Self::new(value.as_f64().unwrap() as u64)
    }
}

And use it in JavaScript (tested with nodejs version 23.7):

async function main() {
    const rln_wasm = require("../pkg");

    let fr = rln_wasm.WasmFr.new(BigInt(123));
    console.log("fr:", fr); // print: 'fr: WasmFr { __wbg_ptr: 1114128 }'
    console.log("fr debug", fr.format_debug());

    console.log("Fr Zero debug", rln_wasm.WasmFr.zero().format_debug());
}

main();

Similarly to C FFI solution, we could create additional structures like:

#[wasm_bindgen]
#[derive(Debug)]
pub struct WasmRLNProofValues {
    y: WasmFr,
    // nullifier: Fr,
    // root: Fr,
    // x: Fr,
    // external_nullifier: Fr,
}

#[wasm_bindgen]
impl WasmRLNProofValues {
    pub fn new(y: WasmFr) -> Self {
        Self {
            y
        }   
    }
    
    pub fn format_debug(&self) -> String {
        format!("{:?}", self)
    }
}

and in JavaScript:

async function main() {
    const rln_wasm = require("../pkg");
    
    let y = rln_wasm.WasmFr.new(BigInt(11));
    let rln_proof_values = rln_wasm.WasmRLNProofValues.new(y);
    console.log("rln_proof_values:", rln_proof_values.format_debug());
}
main();

Notes:

• Marking new functions with #[wasm_bindgen(constructor)] allow to use the standard Javascript declaration like: let fr = new rln_wasm.WasmFr(BigInt(123))

Limitations:

• Cannot (yet) return tuple of struct but returning a Vec<WasmFr> is ok.

Conclusion

The proposed solution has enormous potential to improve Zerokit's FFI as we have shown with the provided examples.

While I'm confident with the examples, the best way to transition, would be to start a POC with a more complete (but still not exhaustive) example like: generating a RLN proof and verifying it in C. It would also be worth checking about how the current examples would work with Nim.

[sydhds]

From DM with @vinhtc27 && @seemenkina, there were questions about how Nim dev in our organisation use a C API (either using directly the C ABI or writing a little wrapper similar to this one for example: https://github.com/arnetheduck/nim-sqlite3-abi/tree/master).

Currently based on this: https://github.com/waku-org/nwaku/blob/master/waku/waku_rln_relay/rln/rln_interface.nim, it seems that this is the first choice (direct use of C ABI).

However it seems like we could auto generate a Nim wrapper based on generated C headers (generated using safer_ffi tooling). Sounds like we start with a POC to see how easy it is.

[Ben-PH]

A couple of thoughts that come to mind:

Referring to "unsafe" in the pejorative

I don't think there is anything inherently wrong with unsafe code. Unsafe scopes are not inherently problematic, and the rhetoric in safer_ffi, specifically in the documetation "without polluting your rust code with unsafe." is a bit jarring to me. FFI is inherently unsafe in nature, and thus requires relatively close attention to verify pre-conditions, account for side effect posibilities, ffi boundary shenanigans, etc. My first impressions of a library that describes the unsafe keyword in an FFI context as "polluting" generates a prejudice in my mind. Making complexity easier to deal with, in this case FFI safety, can either be obfuscated and swept under the rug for developer convenience (e.g. using a garbage collector and related mechanisms to automagically handle memory models), or packaged up in a way that navigating this complexity is made much easier, with a zero-cost to the product (e.g. the borrow checker in Rust: writing a complex system that is memory safe, without a runtime, is much easier in rust than in any other non-runtime languages). I think we will need to carefully investigate the inner dealings of safer_ffi to ensure that using it is a true zero-cost abstraction of FFI complexities.

I suggest looking into the proc-macro expansions such as the #[ffi_export] annotation, and comparing them with how you might write it by hand. Include the highlights/pertinent examples that demonstrate how safer_ffi takes care of the complexities around FFI. My hope is that it is not done "the easy way", and the abstractions are truly zero-cost and of similar/better quality to hand-written solutions.

Bringing in the nim language

The FFI story between C and Rust is well established. I don't see the benefit of thinking about the nim wrappers at this stage. The way I'm thinking of it, is that the Rust/Nim relationship will never exist: it will only ever be Rust/C, and if it just so happens to be used by Nim, then that will be because, after the fact, the C code that uses the FFI will be the result of the nim->C transpilation. Caveate: I'm not too well accomplished with nim to say this with any sort of authority. I do, however, feel suitably qualified to suggest that we initiate progress based on this as a "loosely held" position. This can change where we get input from someone that understands the technical peculiarities of Nim that highlight this as an unwise position to hold.

[sydhds]

I get your point that "unsafe code" is not inherently problematic but that can be an issue for developers not used to deal with FFI.

From what I read (and test) about safer_ffi, here is the advantages:

• Better function signatures (in Rust). There are some great examples in this blog post: https://www.ditto.com/blog/introducing-safer-ffi (see section: How safer_ffi made our function signatures readable)
• Support for type like repr_c::Vec, repr_c::String or repr_c::Box. So you can easily and safely return a Vec of anything (basic type, structure, opaque structure) easily

safer_ffi safety comes from this trait ReprC and this provides some sanity checks.

Here is a basic example showing the expanded code:

#![deny(unsafe_code)] /* No `unsafe` needed! */

use ::safer_ffi::prelude::*;

/// Concatenate two input UTF-8 (_e.g._, ASCII) strings.
///
/// \remark The returned string must be freed with `rust_free_string`
#[ffi_export]
fn concat (fst: char_p::Ref<'_>, snd: char_p::Ref<'_>)
           -> char_p::Box
{
    let fst = fst.to_str(); // : &'_ str
    let snd = snd.to_str(); // : &'_ str
    format!("{}{}", fst, snd) // -------+
        .try_into() //                   |
        .unwrap() // <- no inner nulls --+
}

/// Frees a Rust-allocated string.
#[ffi_export]
fn rust_free_string (string: char_p::Box)
{
    drop(string)
}

and the expanded code:

#![feature(prelude_import)]
#![deny(unsafe_code)]
#[prelude_import]
use std::prelude::rust_2024::*;
#[macro_use]
extern crate std;
use ::safer_ffi::prelude::*;
/// Concatenate two input UTF-8 (_e.g._, ASCII) strings.
///
/// \remark The returned string must be freed with `rust_free_string`
#[allow(improper_ctypes_definitions)]
#[forbid(elided_lifetimes_in_paths)]
extern "C" fn concat(fst: char_p::Ref<'_>, snd: char_p::Ref<'_>) -> char_p::Box {
    {
        /// Concatenate two input UTF-8 (_e.g._, ASCII) strings.
        ///
        /// \remark The returned string must be freed with `rust_free_string`
        #[allow(improper_ctypes_definitions)]
        #[forbid(elided_lifetimes_in_paths)]
        #[export_name = "concat"]
        extern "C" fn concat__ffi_export__(
            fst: <char_p::Ref<'_> as ::safer_ffi::ඞ::ConcreteReprC>::ConcreteCLayout,
            snd: <char_p::Ref<'_> as ::safer_ffi::ඞ::ConcreteReprC>::ConcreteCLayout,
        ) -> <char_p::Box as ::safer_ffi::ඞ::ConcreteReprC>::ConcreteCLayout {
            let abort_on_unwind_guard;
            (
                abort_on_unwind_guard = ::safer_ffi::ඞ::UnwindGuard("concat"),
                unsafe {
                    ::safer_ffi::layout::into_raw(
                        concat(
                            ::safer_ffi::layout::from_raw_unchecked(fst),
                            ::safer_ffi::layout::from_raw_unchecked(snd),
                        ),
                    )
                },
                ::safer_ffi::ඞ::mem::forget(abort_on_unwind_guard),
            )
                .1
        }
    }
    let fst = fst.to_str();
    let snd = snd.to_str();
    ::alloc::__export::must_use({
            ::alloc::fmt::format(format_args!("{0}{1}", fst, snd))
        })
        .try_into()
        .unwrap()
}
/// Frees a Rust-allocated string.
#[allow(improper_ctypes_definitions)]
#[forbid(elided_lifetimes_in_paths)]
extern "C" fn rust_free_string(string: char_p::Box) {
    {
        /// Frees a Rust-allocated string.
        #[allow(improper_ctypes_definitions)]
        #[forbid(elided_lifetimes_in_paths)]
        #[export_name = "rust_free_string"]
        extern "C" fn rust_free_string__ffi_export__(
            string: <char_p::Box as ::safer_ffi::ඞ::ConcreteReprC>::ConcreteCLayout,
        ) -> ::safer_ffi::ඞ::CLayoutOf<()> {
            let abort_on_unwind_guard;
            (
                abort_on_unwind_guard = ::safer_ffi::ඞ::UnwindGuard(
                    "rust_free_string",
                ),
                unsafe {
                    ::safer_ffi::layout::into_raw(
                        rust_free_string(::safer_ffi::layout::from_raw_unchecked(string)),
                    )
                },
                ::safer_ffi::ඞ::mem::forget(abort_on_unwind_guard),
            )
                .1
        }
    }
    drop(string)
}

[Ben-PH]

Seems to me that there is potential for non-zero cost in the unwinding management. I'm not 100% groked on this level of FFI details, but I am at least aware that stack unwinding across FFI is one of the more difficult aspects of FFI. My guess is that when exceptions are mixed into the fray, that just makes the whole landscape spicier.

There's also the mem layout shenanigans to consider. If that could be done at compile time, hence zero cost, then it would not need the unsafe block, at least that's my guess.

My read into this expansion is that there are two particularly interesting aspects that are accomplished here.

1. There is well thought out encapsulation of the "FFI problem" within the type system.
2. It handles (assumedly) all possible cases of unsafe-guards to cut out the use of unsafe to the library user.

my take is that on point 1 alone, safer_ffi leans into a significant aspect of why Rust is recognised as one of the most technically accomplished languages out there.

my take on two is along the lines of "If the unwind guard and memory layout shenanigans are present in all code handwritten by someone that really knows what they are doing, then this looks like a zero-cost abstraction". The question remains in my mind, is to do with that "if", and where there might be overhead (z-cost or not), is it documented in blogposts, documentation, etc? If it is well documented, that implies, to me at least, that the principal of zero cost abstraction in Rust is being respected. A crate that incurs overhead, but discusses why, how, etc., and provides details of opting out of certain costs, clearly demonstrates a direction where, even if zero-cost is not achieved, it is a clear objective of the project to not unreasonable fornc it upon the crate user.