relae | A Relay inspired library for React and RESTful backends | Frontend Framework library

The third party lib (torch) was built using a Pre-cxx11 ABI, and libs built with that couldn't obviously be linked to the object that was using cxx11 ABI!

After hours of debugging of a code that worked just fine in Windows both in Visual Studio and CMake while being a pain in the neck in Linux I found the culprit!

The libtorch is shipped with two types of build Pre-cxx11 and cxx11 ABIs! It was using the Pre-cxx11 built libs that was shipped with its Python package (torch1.6cpu) and since Anaconda3 was in the Path and I was also using it to build the libs I faced this issue.

What gave it away after all these hours was the weird arguments to the undefined methods which were : std::__cxx11::basic_string where it should have been simply std::string. I didn't expect this two to be different and thought, that's a weird naming scheme the g++ is using until moments ago, that made me say, let's give that a search maybe I can get something out it! Lo and behold! this was the case :

If you get linker errors about undefined references to symbols that involve types in the std::__cxx11 namespace or the tag [abi:cxx11] then it probably indicates that you are trying to link together object files that were compiled with different values for the _GLIBCXX_USE_CXX11_ABI macro. This commonly happens when linking to a third-party library that was compiled with an older version of GCC. If the third-party library cannot be rebuilt with the new ABI then you will need to recompile your code with the old ABI.

ref

To fix this I simply directly used the libtorch cxx11 in all libs creation and that just did it. meaning, unlike what is being shown in the Pytorch's official documentation. Do not do:

What is the purpose of each of those four mappings?
Why does the dynamic loader create a "padding" mapping with no permissions?

In order to understand the final state, we need to trace through the actions that dynamic linker takes. What are its "instructions"? It needs to load ET_DYN object in memory, at a random address (chosen by the OS). The mappings must satisfy these "commands" (I omitted PhysAddr, since it's the same as VirtAddr):

If you do a diff of the output of readelf -a, you'll see that there is a change in the "Build ID" field:

Have you explored internals of copy operation for dynamic arrays - _DynArrayCopy in system.pas?

It checks type of all fields of internal data structures, copies simple data with Move, and recursively calls itself for dynamic arrays fields.

Perhaps you final result will be alike this.

On GNU/Linux, almost all Go executables fall into these categories:

1. Those that include the application, the Go run-time, and a statically linked copy of (parts of) glibc.
2. Those that include just the application and the Go run-time, statically linked, and none of glibc.
3. Those that include just the application and the Go run-time, statically linked, and link to glibc dynamically.

Go-related tooling often conflates the two, unfortunately. The main reason for the glibc dependency is that the application uses host name and user lookup (functions like getaddrinfo and getpwuid_r). CGO_ENABLED=0 switches from implementations like src/os/user/cgo_lookup_unix.go (uses glibc) to src/os/user/lookup_unix.go (does not use glibc). The non-glibc implementation does not use NSS and thus offers somewhat limited functionality (which generally do not affect users that do not store user information in LDAP/Active Directory).

In your case, setting CGO_ENABLED=0 moves your application from the third category to the second. (There is other Go-related tooling that could build an application of the first kind.) The non-NSS lookup code is not very large, so the increase in binary size is not significant. Since the Go run-time was already statically linked, it's even possible that the reduced overhead from static linking results in net reduction of executable size.

The most important issue to consider here is that NSS, threads and static linking do not all that well in glibc. All Go programs are multi-threaded, and the reason to (statically) link glibc into Go programs is precisely access to NSS functions. Therefore, statically linking Go programs against glibc is always the wrong thing to do. It is basically always buggy. Even if Go programs were not multi-threaded, a statically linked program which uses NSS functions needs the exact same version of glibc at run time that was used at build time, so static linking of such application reduces portability.

All these are reasons why Go applications of the first kind are such a bad idea. Producing a statically linked application using CGO_ENABLED=0 does not have these problems because those applications (of the second kind) do not include any glibc code (at the cost of reduced functionality of the user/host lookup functions).

If you want to create a portable binary which needs glibc, you need to link your application dynamically (the third kind), on a system with the oldest glibc you want to support. The application will then run on that glibc version and all later versions (for now, Go does not link libc correctly, so there is no strong compatibility guarantuee even for glibc). Distributions are generally ABI-compatible, but they have different versions of glibc. glibc goes to great lengths to make sure that applications dynamically linked against older versions of glibc will keep running on new versions of glibc, but the converse is not true: Once you link an application on a certain version of glibc, it may pick up features (symbols) that are just not available on older versions, so the application will not work with those older versions.

Your entry point attempts to return to a return address on the stack which does not exist, which is why the program jumps to address zero.

The entry point of a program is not expected to return. It must terminate the process by calling _exit (or a related system call).

First of all, it's OK to mix different C++ runtimes in one app, as long as they do not interact. What this means, the C++ objects (including exceptions) should not cross the boundaries of their shared library. Therefore, if your libB.so provides an extern "C" public API, you can safely use these functions from components compiled for any C++ runtime, even stlport_static.

If the vendor lib does not export a pure C API, it's a violation of the architecture provisions (see https://source.android.com/devices/architecture/images/vndk_design_android_o.pdf, page 27). In this situation, you may need to build your dependent libA.so as part of AOSP, too. You can do that if you choose to couple your library more tightly to libB.so, e.g. extend some of its classes, throw exceptions across libA/libB boundary, etc.

Note that since Android N, the system linker protects platform libraries (those in /system/lib and /vendor/lib) from being dlopened from user code. You should consider adding libB, or libA, or both, to the whitelist.

ET_DYN is used for position-independent executables (PIE), whether they are statically linked or not. The lack of a program interpreter and DT_NEEDED entries in the dynamic section indicate that the program is indeed statically linked. You can check that using readelf -l (no .interp) and readelf -d (no NEEDED).

Running the program under strace will also verify that no shared objects are loaded at program start.