About Blog Contact

Strange Symbol Duplication in Dynamic Library

I am writing a PyTorch C++ extension, and a very peculiar problem came up. PyTorch’s C++ extension system allows one to implement a nn.Moduleor a autograd function in C++. Using the pybind11 library, the extension system compiles the C++ modules or functions into a .so dynamic library, and the Python interpreter loads it during launch, exposing it as just another Python object. With this feature, one can write the timing critical code in C++ for speed, while writing the other code in Python for ease of development.

However, during my development, I found out that the Python script fails at launch with an error of missing symbols in the .so library. However, when I grepped the nm -C symbol list of the .so file, the symbol is right there:

U projectNamespace::someFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const
0000000000391b10 t projectNamespace::someFunction(std::string const&) const                                                                          

The U here means the symbol is undefined, while the t means that the symbol is in the text section. The two symbol are actually the same: std::string is actually a instantiation of a class template. In header <string>, we can see:

template<
    class CharT,
    class Traits = std::char_traits<CharT>,
    class Allocator = std::allocator<CharT>
> class basic_string;
using string = basic_string<char>; 

However, this duplication somehow confuses the Python interpreter, making it fail to find the function when it’s right there.

I solved this problem by changing cmake tool to anaconda’s version, therefore changing the g++ compiler for the C++ extension. Now, the symbol can be found by Python, and the undefined symbol disappears:

0000000000391b10 t projectNamespace::someFunction(std::string const&) const

All of this is very weird to me, but unfortunately, I do not have the time to investigate further, as I’m already very behind on schedule, and my boss is getting agitated. So I’ll just leave it there for now.