Strange Symbol Duplication in Dynamic Library
I am writing a PyTorch C++ extension, and a very peculiar problem came up. PyTorch’s C++ extension system allows one to implement a nn.Module
or a autograd function in C++. Using the pybind11
library, the extension system compiles the C++ modules or functions into a .so
dynamic library, and the Python interpreter loads it during launch, exposing it as just another Python object. With this feature, one can write the timing critical code in C++ for speed, while writing the other code in Python for ease of development.
However, during my development, I found out that the Python script fails at launch with an error of missing symbols in the .so
library. However, when I grepped the nm -C
symbol list of the .so
file, the symbol is right there:
U projectNamespace::someFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const
0000000000391b10 t projectNamespace::someFunction(std::string const&) const
The U
here means the symbol is undefined, while the t
means that the symbol is in the text section. The two symbol are actually the same: std::string
is actually a instantiation of a class template. In header <string>
, we can see:
template<
class CharT,
class Traits = std::char_traits<CharT>,
class Allocator = std::allocator<CharT>
> class basic_string;
using string = basic_string<char>;
However, this duplication somehow confuses the Python interpreter, making it fail to find the function when it’s right there.
I solved this problem by changing cmake
tool to anaconda’s version, therefore changing the g++
compiler for the C++ extension. Now, the symbol can be found by Python, and the undefined symbol disappears:
0000000000391b10 t projectNamespace::someFunction(std::string const&) const
All of this is very weird to me, but unfortunately, I do not have the time to investigate further, as I’m already very behind on schedule, and my boss is getting agitated. So I’ll just leave it there for now.