Error in Questa simulator when importing AES from Crypto.Cipher

[yuzhang92]

I saw a similar issue w.r.t "Getting an error in Questa simulator when importing PyCrypto library from Python #16"

The python I used is python 3.6.5 and i have installed pycryptodome.

I am using linux platform and followed the setup, but it seems importing AES from Crypto.Cipher gives me errors when running it with questasim (ver 2020.4):

# do run.do
terminate called after throwing an instance of 'pybind11::error_already_set'
what(): AttributeError: module 'Crypto' has no attribute 'Cipher'

My code looks like below:
[crypto_test.py]

from Crypto.Cipher import AES
from pysv import sv, compile_lib, generate_sv_binding, DataType

@sv(msg=DataType.String,key=DataType.String,return_type=DataType.String)
def crypto_algo(msg,key):
    import hashlib
    import hmac
    import binascii
    cipher = AES.new(key, AES.MODE_ECB)  # whenever i have this line, it gives error, which blocks me from creating any AES cipher
    key = key.encode('utf-8')
    return_data = hashlib.sha256(msg.encode('utf-8')).hexdigest()
    print("msg =")
    print(msg)
    print("key =")
    print(key)
    print("return_data =")
    print(return_data)

    return return_data

lib_path = compile_lib([crypto_algo], cwd="build")
generate_sv_binding([crypto_algo], filename="crypto_algo_pkg.sv")

[top1.sv]

module top1();
    import pysv::crypto_algo;
    int check;
    string str = "";
    initial begin
        check = 0;
        str = crypto_algo("0123456789012345","1122334455667788");
        $display("res = %s", str);
    end
endmodule

[vsim.sh]

rm -rf work
vlib work

vlog crypto_algo_pkg.sv top1.sv
vopt top1 +acc -l opt.log  -o top1_opt

vsim top1_opt -sv_lib build/libpysv -batch -do run.do

[run.do]
run -all

[Kuree]

Can you try to import Crypto first? e.g. import Crypto at the beginning?

I think there is some issue in pysv that not be able to pick up foreign module. I will fix this issue.

[yuzhang92]

Can you try to import Crypto first? e.g. import Crypto at the beginning?

I think there is some issue in pysv that not be able to pick up foreign module. I will fix this issue.

Thanks for the quick reply, updated trial with below coding:

import Crypto
from Crypto.Cipher import AES
from pysv import sv, compile_lib, generate_sv_binding, DataType

@sv(msg=DataType.String,key=DataType.String,return_type=DataType.String)
def crypto_algo(msg,key):
    import hashlib
    import hmac
    import binascii
    cipher = AES.new(key, AES.MODE_ECB)
    key = key.encode('utf-8')
    return_data = hashlib.sha256(msg.encode('utf-8')).hexdigest()
    print("msg =")
    print(msg)
    print("key =")
    print(key)
    print("return_data =")
    print(return_data)

    return return_data

lib_path = compile_lib([crypto_algo], cwd="build")
generate_sv_binding([crypto_algo], filename="crypto_algo_pkg.sv")

The issue is still there on my side:

# do run.do
terminate called after throwing an instance of 'pybind11::error_already_set'
what(): AttributeError: module 'Crypto' has no attribute 'Cipher'

[Kuree]

It looks like the module is imported correctly, but the submodule is not. Can you copy and paste the generated C++ code here? It should have the python path hardcoded in. You can mask off username etc.

[yuzhang92]

It looks like the module is imported correctly, but the submodule is not. Can you copy and paste the generated C++ code here? It should have the python path hardcoded in. You can mask off username etc.

[libpysv.cc]

#include "pybind11/include/pybind11/embed.h"
#include "pybind11/include/pybind11/eval.h"
#include <iostream>
#include <unordered_map>
#include <memory>

// used for ModelSim/Questa to resolve some runtime native library loading issues
// not needed for Xcelium and vcs, but include just in case
#ifdef __linux__
#include <dlfcn.h>
#endif
namespace py = pybind11;
std::unique_ptr<std::unordered_map<std::string, py::object>> global_imports = nullptr;
std::unique_ptr<py::scoped_interpreter> guard = nullptr;
std::unique_ptr<std::unordered_map<void*, py::object>> py_obj_map;
std::unique_ptr<py::dict> class_defs;
std::string string_result_value;

std::string conda_python_home;
std::string conda_python_path;
std::string get_env(const char *name) {
    std::string result;
#ifdef _WIN32
    char *path_var;
    size_t len;
    auto err = _dupenv_s(&path_var, &len, name);
    if (err) {
        env_path = "";
    }
    result = std::string(path_var);
    free(path_var);
    path_var = nullptr;
#else
    auto r = std::getenv(name);
    if (r) {
        result = r;
    }
#endif
    return result;
}

void unset_env(const char *name) {
    unsetenv(name);
}

std::pair<std::string, std::string> get_py_env() {
    std::string python_home = get_env("PYTHONHOME");
    std::string python_path = get_env("PYTHONPATH");
    return std::make_pair(python_home, python_path);
}

void unset_py_env() {
    unset_env("PYTHONHOME");
    unset_env("PYTHONPATH");
}

void set_env(const char *name, const std::string &value) {
#ifdef _WIN32
    _putenv_s(name, value.c_str());
#else
    setenv(name, value.c_str(), 1);
#endif
}

void set_py_env(const std::pair<std::string, std::string> values) {
    set_env("PYTHONHOME", values.first);
    set_env("PYTHONPATH", values.second);
}

static bool has_py_env_set = false;
void initialize_guard() {
    if (guard) return;
    // make sure if PYTHONHOME and PYTHONPATH are set or not
    auto python_env_vars = get_py_env();
    // if it is set, clear it out temporally and then restore it later
    // when we check the system path
    if (python_env_vars.first.empty()) {
        // we all good
        // can't use make_unique since it's c++14 only
        if (!conda_python_home.empty()) {
            set_py_env(std::make_pair(conda_python_home, conda_python_path));
        }
        guard = std::unique_ptr<py::scoped_interpreter>(new py::scoped_interpreter());
    } else {
        // unset the env
        unset_py_env();
        guard = std::unique_ptr<py::scoped_interpreter>(new py::scoped_interpreter());
        // then restore it
        set_py_env(python_env_vars);
        has_py_env_set = true;
    }
}
auto SYS_PATH = {"/home/<username>/experiment/pysv_example/pycrypto",
                 "/designtools/python_3.6.5/lib/python36.zip",
                 "/designtools/python_3.6.5/lib/python3.6",
                 "/designtools/python_3.6.5/lib/python3.6/lib-dynload",
                 "/home/<username>/.local/lib/python3.6/site-packages",
                 "/designtools/python_3.6.5/lib/python3.6/site-packages",
                 "/designtools/python_3.6.5/lib/python3.6/site-packages/CryptoPlus-1.0-py3.6.egg"};

void check_sys_path(const char *python_lib) {
    // always check guard first
    initialize_guard();
    // only modify the sys.path if the one is . (set by pybind)
    auto sys = py::module::import("sys");
    auto sys_path = sys.attr("path");
    auto last_path_len = py::len(sys.attr("path").attr("__getitem__")(py::int_(-1)));
    if (last_path_len == 1) {
        // need to set the path
        // clear first
        if (!has_py_env_set) {
            sys.attr("path").attr("clear")();
        }

        for (auto const &path: SYS_PATH) {
            sys.attr("path").attr("append")(py::str(path));
        }
        // also load it into the global table if it's on linux. only needed for ModelSim/Questa
        #ifdef __linux__
        dlopen(python_lib, RTLD_LAZY | RTLD_GLOBAL);
        #endif
    }
}
std::vector<std::string> get_tokens(const std::string &line, const std::string &delimiter) {
    std::vector<std::string> tokens;
    size_t prev = 0, pos = 0;
    std::string token;
    // copied from https://stackoverflow.com/a/7621814
    while ((pos = line.find_first_of(delimiter, prev)) != std::string::npos) {
        if (pos > prev) {
            tokens.emplace_back(line.substr(prev, pos - prev));
        }
        prev = pos + 1;
    }
    if (prev < line.length()) tokens.emplace_back(line.substr(prev, std::string::npos));
    // remove empty ones
    std::vector<std::string> result;
    result.reserve(tokens.size());
    for (auto const &t : tokens)
        if (!t.empty()) result.emplace_back(t);
    return result;
}

void import_module(const std::string &module_name, const std::string &imported_name,
                   py::dict &globals) {
    if (!global_imports) {
        global_imports = std::unique_ptr<std::unordered_map<std::string, py::object>>(new std::unordered_map<std::string, py::object>());
    }
    if (global_imports->find(module_name) == global_imports->end()) {
        py::object target;
        // tokenize the module name in case it has nested namespace
        bool top_module = true;
        auto name_tokens = get_tokens(module_name, ".");
        for (auto const &name: name_tokens) {
            if (top_module) {
                target = py::module::import(name.c_str());
                top_module = false;
            } else {
                target = target.attr(name.c_str());
            }
        }
        global_imports->emplace(module_name, target);
    }
    auto &m = global_imports->at(module_name);
    globals[imported_name.c_str()] = m;
}

extern "C" {
__attribute__((visibility("default"))) const char* crypto_algo(const char* msg,
                                                               const char* key) {
  check_sys_path(PYTHON_LIBRARY);
  auto globals = py::dict();
  import_module("Crypto", "Crypto", globals);
  import_module("Crypto.Cipher.AES", "AES", globals);

  auto locals = py::dict();
  locals["__msg"] = msg;
  locals["__key"] = key;

  py::exec(R"(
def crypto_algo(msg, key):
    import hashlib
    import hmac
    import binascii
    cipher = AES.new(key, Crypto.Cipher.AES.MODE_ECB)
    key = key.encode('utf-8')
    return_data = hashlib.sha256(msg.encode('utf-8')).hexdigest()
    print('msg =')
    print(msg)
    print('key =')
    print(key)
    print('return_data =')
    print(return_data)
    return return_data

__result = crypto_algo(__msg, __key)
)", globals, locals);
  string_result_value = locals["__result"].cast<std::string>();
  return string_result_value.c_str();
}
__attribute__((visibility("default"))) void pysv_finalize() {
    // clear the cached global imports
    global_imports.reset();
    // clear the object map
    py_obj_map.reset();
    // clear the class map
    class_defs.reset();
    // the last part is tear down the runtime
    guard.reset();
}
}

One thing I notice is that the Crypto package from my side is being installed in a virtual env (venv) path, which is located at /home//pyvenv/env3/lib/python3.6/site-packages/*, e.g.

pwd: /home/<username>/pyvenv/env3/lib/python3.6/site-packages

[[site-packages]$ tree -L 1
├── astor
├── astor-0.8.1.dist-info
├── Crypto    => this is the installed package for Crypto I want 
├── easy_install.py
├── numpy
├── numpy-1.19.5.dist-info
├── numpy.libs
├── pip
├── pip-9.0.3.dist-info
├── pkg_resources
├── __pycache__
├── pycryptodome-3.15.0.dist-info
├── pysv
├── pysv-0.2.0.dist-info
├── setuptools
└── setuptools-39.0.1.dist-info
├── numpy.libs
├── pip
├── pip-9.0.3.dist-info
├── pkg_resources
├── __pycache__
├── pycryptodome-3.15.0.dist-info
├── pysv
├── pysv-0.2.0.dist-info
├── setuptools
└── setuptools-39.0.1.dist-info

Looks like the generated libpysv.cc does not include that venv path. But it includes this path:

"/home//.local/lib/python3.6/site-packages", which also includes the Crypto/Cipher/....

[Kuree]

Just to make sure, you ran pysv inside the venv environment right?

The python path is generated from this code:

pysv/pysv/codegen.py

Lines 474 to 485 in 3c8f05b

	def generate_sys_path_values(pretty_print=True):
	result = "auto " + __SYS_PATH_NAME + " = {"
	if pretty_print:
	padding = ",\n" + len(result) * " "
	else:
	padding = ", "
	path_values = []
	for p in sys.path:
	path_values.append('"{0}"'.format(p))
	result += padding.join(path_values)
	result += "};\n\n"
	return result

Does your sys.path match with the output from pysv?

EDIT:
Here is my sys.path output inside the venv:

$ python
Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/<username>/workspace/pysv/env/lib/python3.10/site-packages', '/home/<username>/workspace/pysv']

[yuzhang92]

Just to make sure, you ran pysv inside the venv environment right?

The python path is generated from this code:

pysv/pysv/codegen.py

Lines 474 to 485 in 3c8f05b

	def generate_sys_path_values(pretty_print=True):
	result = "auto " + __SYS_PATH_NAME + " = {"
	if pretty_print:
	padding = ",\n" + len(result) * " "
	else:
	padding = ", "
	path_values = []
	for p in sys.path:
	path_values.append('"{0}"'.format(p))
	result += padding.join(path_values)
	result += "};\n\n"
	return result

Does your sys.path match with the output from pysv?

EDIT: Here is my sys.path output inside the venv:

$ python
Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/<username>/workspace/pysv/env/lib/python3.10/site-packages', '/home/<username>/workspace/pysv']

i noticed previously there were some issues in my venv setup, now I have re-setup & re-run it with venv and this is the SYS_PATH:

auto SYS_PATH = {"/home/<username>/_experiment/pysv_example/pycrypto",
                 "/designtools/python_3.6.5/lib/python36.zip",
                 "/designtools/python_3.6.5/lib/python3.6",
                 "/designtools/python_3.6.5/lib/python3.6/lib-dynload",
                 "/home/<username>/pyvenv/env3/lib/python3.6/site-packages"};

My sys.path looks like this:

[env3] [pycrypto]$ python
Python 3.6.5 (default, Dec 19 2019, 11:57:42) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/designtools/python_3.6.5/lib/python36.zip', '/designtools/python_3.6.5/lib/python3.6', '/designtools/python_3.6.5/lib/python3.6/lib-dynload', '/home/<username>/pyvenv/env3/lib/python3.6/site-packages']

From what I can see it matches

The ERROR msg "AttributeError: module 'Crypto' has no attribute 'Cipher'" from questasim still exists with this run.

====================================

full libpysv.cc

#include "pybind11/include/pybind11/embed.h"
#include "pybind11/include/pybind11/eval.h"
#include <iostream>
#include <unordered_map>
#include <memory>

// used for ModelSim/Questa to resolve some runtime native library loading issues
// not needed for Xcelium and vcs, but include just in case
#ifdef __linux__
#include <dlfcn.h>
#endif
namespace py = pybind11;
std::unique_ptr<std::unordered_map<std::string, py::object>> global_imports = nullptr;
std::unique_ptr<py::scoped_interpreter> guard = nullptr;
std::unique_ptr<std::unordered_map<void*, py::object>> py_obj_map;
std::unique_ptr<py::dict> class_defs;
std::string string_result_value;

std::string conda_python_home;
std::string conda_python_path;
std::string get_env(const char *name) {
    std::string result;
#ifdef _WIN32
    char *path_var;
    size_t len;
    auto err = _dupenv_s(&path_var, &len, name);
    if (err) {
        env_path = "";
    }
    result = std::string(path_var);
    free(path_var);
    path_var = nullptr;
#else
    auto r = std::getenv(name);
    if (r) {
        result = r;
    }
#endif
    return result;
}

void unset_env(const char *name) {
    unsetenv(name);
}

std::pair<std::string, std::string> get_py_env() {
    std::string python_home = get_env("PYTHONHOME");
    std::string python_path = get_env("PYTHONPATH");
    return std::make_pair(python_home, python_path);
}

void unset_py_env() {
    unset_env("PYTHONHOME");
    unset_env("PYTHONPATH");
}

void set_env(const char *name, const std::string &value) {
#ifdef _WIN32
    _putenv_s(name, value.c_str());
#else
    setenv(name, value.c_str(), 1);
#endif
}

void set_py_env(const std::pair<std::string, std::string> values) {
    set_env("PYTHONHOME", values.first);
    set_env("PYTHONPATH", values.second);
}

static bool has_py_env_set = false;
void initialize_guard() {
    if (guard) return;
    // make sure if PYTHONHOME and PYTHONPATH are set or not
    auto python_env_vars = get_py_env();
    // if it is set, clear it out temporally and then restore it later
    // when we check the system path
    if (python_env_vars.first.empty()) {
        // we all good
        // can't use make_unique since it's c++14 only
        if (!conda_python_home.empty()) {
            set_py_env(std::make_pair(conda_python_home, conda_python_path));
        }
        guard = std::unique_ptr<py::scoped_interpreter>(new py::scoped_interpreter());
    } else {
        // unset the env
        unset_py_env();
        guard = std::unique_ptr<py::scoped_interpreter>(new py::scoped_interpreter());
        // then restore it
        set_py_env(python_env_vars);
        has_py_env_set = true;
    }
}
auto SYS_PATH = {"/home/yuzhang/tc_verif22_experiment/pysv_example/pycrypto",
                 "/designtools/python_3.6.5/lib/python36.zip",
                 "/designtools/python_3.6.5/lib/python3.6",
                 "/designtools/python_3.6.5/lib/python3.6/lib-dynload",
                 "/home/yuzhang/pyvenv/env3/lib/python3.6/site-packages"};

void check_sys_path(const char *python_lib) {
    // always check guard first
    initialize_guard();
    // only modify the sys.path if the one is . (set by pybind)
    auto sys = py::module::import("sys");
    auto sys_path = sys.attr("path");
    auto last_path_len = py::len(sys.attr("path").attr("__getitem__")(py::int_(-1)));
    if (last_path_len == 1) {
        // need to set the path
        // clear first
        if (!has_py_env_set) {
            sys.attr("path").attr("clear")();
        }

        for (auto const &path: SYS_PATH) {
            sys.attr("path").attr("append")(py::str(path));
        }
        // also load it into the global table if it's on linux. only needed for ModelSim/Questa
        #ifdef __linux__
        dlopen(python_lib, RTLD_LAZY | RTLD_GLOBAL);
        #endif
    }
}
std::vector<std::string> get_tokens(const std::string &line, const std::string &delimiter) {
    std::vector<std::string> tokens;
    size_t prev = 0, pos = 0;
    std::string token;
    // copied from https://stackoverflow.com/a/7621814
    while ((pos = line.find_first_of(delimiter, prev)) != std::string::npos) {
        if (pos > prev) {
            tokens.emplace_back(line.substr(prev, pos - prev));
        }
        prev = pos + 1;
    }
    if (prev < line.length()) tokens.emplace_back(line.substr(prev, std::string::npos));
    // remove empty ones
    std::vector<std::string> result;
    result.reserve(tokens.size());
    for (auto const &t : tokens)
        if (!t.empty()) result.emplace_back(t);
    return result;
}

void import_module(const std::string &module_name, const std::string &imported_name,
                   py::dict &globals) {
    if (!global_imports) {
        global_imports = std::unique_ptr<std::unordered_map<std::string, py::object>>(new std::unordered_map<std::string, py::object>());
    }
    if (global_imports->find(module_name) == global_imports->end()) {
        py::object target;
        // tokenize the module name in case it has nested namespace
        bool top_module = true;
        auto name_tokens = get_tokens(module_name, ".");
        for (auto const &name: name_tokens) {
            if (top_module) {
                target = py::module::import(name.c_str());
                top_module = false;
            } else {
                target = target.attr(name.c_str());
            }
        }
        global_imports->emplace(module_name, target);
    }
    auto &m = global_imports->at(module_name);
    globals[imported_name.c_str()] = m;
}

extern "C" {
__attribute__((visibility("default"))) const char* crypto_algo(const char* msg,
                                                               const char* key) {
  check_sys_path(PYTHON_LIBRARY);
  auto globals = py::dict();
  import_module("Crypto.Cipher.AES", "AES", globals);

  auto locals = py::dict();
  locals["__msg"] = msg;
  locals["__key"] = key;

  py::exec(R"(
def crypto_algo(msg, key):
    import hashlib
    import hmac
    import binascii
    cipher = AES.new(key, AES.MODE_ECB)
    key = key.encode('utf-8')
    return_data = hashlib.sha256(msg.encode('utf-8')).hexdigest()
    print('msg =')
    print(msg)
    print('key =')
    print(key)
    print('return_data =')
    print(return_data)
    return return_data

__result = crypto_algo(__msg, __key)
)", globals, locals);
  string_result_value = locals["__result"].cast<std::string>();
  return string_result_value.c_str();
}
__attribute__((visibility("default"))) void pysv_finalize() {
    // clear the cached global imports
    global_imports.reset();
    // clear the object map
    py_obj_map.reset();
    // clear the class map
    class_defs.reset();
    // the last part is tear down the runtime
    guard.reset();
}
}

[Kuree]

Hmm, looks like the import logic might not be correct. I can try to reproduce it on my end. What is the pypi package name for that module?

I think one workaround is to import the module name only, and then use the full name in your code. Maybe that will work? e.g.

import Crypto

# later on inside the function
from Crypto.Cipher import AES

[yuzhang92]

Hmm, looks like the import logic might not be correct. I can try to reproduce it on my end. What is the pypi package name for that module?

I think one workaround is to import the module name only, and then use the full name in your code. Maybe that will work? e.g.

import Crypto

# later on inside the function
from Crypto.Cipher import AES

The pypi module is "pycryptodome"
you can use
pip install pycryptodome

As for the workaround are you suggesting something like this?

import Crypto
import base64
from pysv import sv, compile_lib, generate_sv_binding, DataType

@sv(msg=DataType.String,key=DataType.String,return_type=DataType.String)
def crypto_algo(msg,key):
    import hashlib
    import hmac
    import binascii
    from base64 import b64decode
    from base64 import b64encode
    from Crypto.Cipher import AES
    cipher = AES.new(key.encode('utf-8'), AES.MODE_ECB)
    return_data = cipher.encrypt(msg.encode('utf8'))
    return_data = b64encode(return_data)
    #return_data = hashlib.sha256(msg.encode('utf-8')).hexdigest()
    print("msg =")
    print(msg)
    print("key =")
    print(key)
    print("return_data =")
    print(return_data)

    return return_data

lib_path = compile_lib([crypto_algo], cwd="build")
generate_sv_binding([crypto_algo], filename="crypto_algo_pkg.sv")

I had a quick trial on the above code and it seems the import issue is gone.

vsim log:

# Loading sv_std.std
# Loading work.pysv(fast)
# Loading work.top1(fast)
# Loading ./build/libpysv.so
# 
# do run.do
msg =
0123456789012345
key =
1122334455667788
return_data =
b'FkE5E56yuopSRShJler0yw=='
# res = FkE5E56yuopSRShJler0yw==
VSIM 2> 

[Kuree]

The pypi module is "pycryptodome"

Thanks. I will take a look and fix the import logic.

As for the workaround are you suggesting something like this?

Yes this is exactly what I have in mind.

I'm preparing my defense and dissertation right now so I'm not sure how much bandwidth I have to fix this issue. I will leave this issue open to track it.

Thanks for reporting it and providing good examples.