Shared libraries that depend on each other?

WARNING: this question might very well have not a lot to do with Racket. We'll see.

I'm working with a student, Alex MacLean, who's developing racket bindings for OpenFST. You can see the current version on the package manager, actually. Since this is a C++ library, name-mangling make straightforward use of the shared library ... difficult? ... Alex has chosen to create a separate library that re-exports these bindings with simpler names. If these are both shared libraries, then it's not clear how to "tell one of them where the other one is". This is in fact a very general question about shared libraries in Linux (and probably on all other platforms as well): is there a standard way for dynamic libraries to depend on each other? Or is the standard approach to re-link the shared library into a new, larger shared library? Or is there some other solution entirely?

3 Likes

I am emphatically not an expert—and I know especially little about C++—but I can try to give a few pointers to things as I understand them, and maybe someone will explain everything I'm wrong about.

One way for dynamic libraries to depend on each other is for the client code to dlopen some library and access its exports using dlsym. That's basically what ffi-lib and get-ffi-obj do. My impression is that it's not really feasible to dlopen C++ code (given name mangling, classes, templates, etc.), which is why wrapper libraries exist: they provide an extern "C" interface, so Racket (or anyone) can dlopen the wrapper, and all the C++ complexity happens where it can be managed by the C++ compiler.

So, if your wrapper library needs to not dlopen the underlying C++ library, it will be loaded implicitly by the dynamic linker. On Linux, that's /lib/ld-linux.so.2:

https://man7.org/linux/man-pages/man8/ld-linux.so.8.html

Basically, it boils down to two strategies:

  • The ELF file for your wrapper library may embed places to look for its dependencies in a DT_RUNPATH (or DT_RPATH) entry (see rpath - Wikipedia). These entries may use special tokens like $ORIGIN to define relative paths. The patchelf utility (GitHub - NixOS/patchelf: A small utility to modify the dynamic linker and RPATH of ELF executables) is a nice way to inspect and modify these entries.

  • Otherwise, the dynamic linker will look in some general system search path, which might involve built-in default locations like /lib and /usr/lib or environment variables like LD_LIBRARY_PATH.

Mac OS has vaguely similar mechanisms I understand in less detail. Using install_name_tool and llvm-objdump can help to inspect or modify some of them. I think @loader_path is analogous to $ORIGIN, and man 1 dyld is the best documentation I know of for the dynamic linker.

Racket can manipulate the $ORIGIN and @loader_path of libraries it installs, e.g.raco setup in response to a copy-foreign-libs definition.

I don't really understand Windows. The only Windows programming I've done since a little Visual Basic 6 a very long time ago is trying to keep my Racket code reasonably portable.

Here is a tangentially interesting blog post about ld-linux.so.2: Taming the ‘stat’ storm with a loader cache — 2021 — Blog — GNU Guix

If you want to distribute the wrapper shared library as a Racket package but use a system-provided OpenFST, I think you may have to just hope that it ends up being in the system's search path. If you are also distributing OpenFST with Racket, I think you want to build your wrapper library so that its DT_RUNPATH finds OpenFST relative to $ORIGIN.

I would love to find a more streamlined way to build shared libraries for use by Racket. This has in fact been on my mind today: my best attempt is at GitHub - LiberalArtist/native-libgit2-pkgs at build-scripts (see the extensive README.md), which works, but I'd like it to be much more convenient. I find Guix particularly tantalizing for this, since it already has definitions for how to build a whole GNU/Linux distribution's worth of software (in a Scheme DSL, even!). Now that I've got it working for libgit2, I'm planning to write up an email to the Guix list with thoughts and questions about how to make this sort of thing work more smoothly.

The build scripts in racket/racket/src/native-libs at master · racket/racket · GitHub are a useful reference, as is Dynext: Running a C Compiler/Linker (also useful for compiling C that is not a BC extension, as in bcrypt.rkt/install.rkt at master · samth/bcrypt.rkt · GitHub).

5 Likes

An even zanier pipe dream I've sometimes entertained is that we could embed a WASM runtime into Racket, then compile many kinds of foreign libraries to WASM, rather than machine code, avoiding all of the platform-specific issues while also reducing the surface area for memory unsafety.

Apparently Firefox has started doing something like this for some libraries: Securing Firefox with WebAssembly - Mozilla Hacks - the Web developer blog

Probably the simplest way would be to write bindings for Wasmtime, but I could also imagine a Nanopass-based WASM front-end for the Chez Scheme compiler.

As with everything WASM, it seems like procrastination would only make things easier, e.g. as the WebAssembly System Interface stabilizes. It's not something I have any plan to work on, at any rate.

3 Likes

I am not sure about all the details and possible gotchas, I only read about it what I needed to get it to work for my case. I think the first thing you should check is that the wrapper library is actually dynamically linked to the wrapped library. (I write linux commands because that is the only platform I currently use)

In my case I am wrapping the freetype library with a small wrapper that makes it easier to extract gylphs (with an api that is easier for me to work with).
So when you use a command like ldd libyourwrapper.so (also try ldd -v ... for more output) you should get a list of dynamic dependencies that includes your wrapped library and also its dependents. If that list does not show those libraries then maybe you are missing some (library) flags while you are building your wrapper library.

I can't give you a general guide towards those flags (I only know what I use, the rest is vague; also other people may know better ways to do things, however I hope it is useful anyway), here is what I use to build my wrapper library:

gcc $(pkg-config --cflags --libs freetype2) -Lfreetype -Wall -fPIC -c glyphloader.c
gcc -shared -Wl,-soname,libglyphloader.so -lfreetype -o libglyphloader.so glyphloader.o

The first line creates the object file, the second creates the shared library out of it.
In my case the output from ldd libglyphloader.so is:

linux-vdso.so.1 (0x00007ffe339eb000)
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x00007f6f7270d000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f6f72500000)
libz.so.1 => /usr/lib/libz.so.1 (0x00007f6f724e6000)
libbz2.so.1.0 => /usr/lib/libbz2.so.1.0 (0x00007f6f724d3000)
libpng16.so.16 => /usr/lib/libpng16.so.16 (0x00007f6f7249c000)
libharfbuzz.so.0 => /usr/lib/libharfbuzz.so.0 (0x00007f6f723ac000)
libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0x00007f6f7239c000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007f6f72823000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f6f722b5000)
libgraphite2.so.3 => /usr/lib/libgraphite2.so.3 (0x00007f6f72290000)
libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x00007f6f72152000)
libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0x00007f6f7212f000)
libpcre.so.1 => /usr/lib/libpcre.so.1 (0x00007f6f720b6000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f6f720b1000)

When I first tried getting this to work I think I didn't have the -lfreetype in the second line and the ldd output was:

linux-vdso.so.1 (0x00007ffea517d000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007fe84817c000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007fe8483d0000)

With that missing loading the wrapper library from racket fails with an error message (... inserted by me):

ffi-lib: could not load foreign library
  path: /home/sze/development/.../libglyphloader.so
  system error: /home/sze/development/.../libglyphloader.so: undefined symbol: FT_Done_Face
  context...:
   /usr/share/racket/collects/ffi/unsafe.rkt:131:0: get-ffi-lib
   body of "/home/sze/development/.../glyphloader.rkt"

Hopefully it is a simple error like this and this information helps.
If it is something more complicated you should look into using other tools that can provide insights, like @LiberalArtist suggested, I think there are a lot of tools, but I only know of a few.

In cases like this everything that can give us hints towards what actually is happening is helpful.
Do you / the student get any kind of error when trying to load the library?

I think it is a standard feature of dynamic libraries to depend on other dynamic libraries, but as far as I know every compiler has different ways of being told to add that link information for those other libraries into the created library file, so they end up as dependencies in the shared libraries. Basically with c and c++ you have to tell the compiler what it should do, because there is no proper module system that keeps track of the dependencies. I imagine that there are other languages / build-systems that automatically manage this.

Another thing is, that with dynamic libraries the actual dynamic linking of the other dependencies only happens when the library is loaded, so that is why you can build a library, that looks as if it compiled correctly, but actually has missing dependencies that you only discover when you try to load it.

If you don't get any error (which seems surprising to me) it may make sense to use a debugger (I mean something like gdb or lldb here, not the racket one, but this mostly if racket doesn't give any insight/error message) to start the racket program that tries to load the library and see whether that can give any insight into what is happening and where the program fails.

But take all of this with a grain of salt, the last time I read something more in depth about this stuff was probably something like 15 years ago. So there may be a lot I have forgotten, or different new tools to handle things like this.

1 Like

As @simonls says above, I think the standard approach is to link your wrapper library against openFST. With a standard C/C++ app, as long as openFST is installed on the target system at runtime, you should be good to go. Modifying the rpath and such isn't normally necessary. I'm not sure what complications result when using the library from a Racket app though.

If the library links fine but doesn't load, i.e. there is a runtime error loading openFST for example, on Linux you can use strace and look for open syscalls. You should see it attempt to load the library from various places. That may give you a clue what went wrong.

1 Like

FWIW here are some useful tools when you are on macOS (quoting [1]):

After you have built a library you can check that it links to the correct libraries with otool:

otool -L foo.dylib

The command will show the paths of the libaries that foo.dylib depends on.
If you see any /opt/local paths here, you have made a mistake (you have linked with a macport library).

During the build the dylibs in ${BuildRacketLibs}/lib will contain full paths.
If you need to distribute the libraries, they need to be changed to relative paths with install_name_tool .

I would do the following:

  1. Place both shared libraries in the same folder
  2. Add the folder to the environment variable controlling the paths used when linking.
  3. Link using -lfoo [where foo.so is the name of the shared library]
  4. Use otool to check that the path is correct inside the shared library.

For more information on shared libraries on Linux, I can't recommend [2] enough.

[1] GitHub - soegaard/racket-osx-libs: Source files and build instructions for the dynamic libraries used by Racket on OS X
[2] https://www.cs.dartmouth.edu/~sergey/cs258/ABI/UlrichDrepper-How-To-Write-Shared-Libraries.pdf

1 Like

I took a look at the specific issue at hand with OpenFST, and the problem was that the installed filename of libfst.so didn't match the RPATH DT_NEEDED of openfst_wrapper.so, which expected libfst.so.25: fix library loading by LiberalArtist · Pull Request #1 · AlexMaclean/racket-openfst · GitHub

2 Likes

Okay, well that was way way above and beyond. You'd better watch out, you're going to get a reputation for fixing people's code gratis! :slight_smile:

Yes, I love this idea. That's exactly what wasm was designed for.

Of course, having said that, the killer app is self-contained computation-heavy code; the more interdependencies you have, the more this works only if everyone else has already ported all of their other stuff, too.

I was reminded of this today when I stumbled across this paper, which (among many other things) presents a compiler from wasm to safe Rust with good performance:

https://www.usenix.org/conference/usenixsecurity22/presentation/bosamiya

It's an interesting paper in a number of respects, and some of the suggested techniques for compiling wasm to a high-level language might be applicable to Chez, but what particularly struck me was that they say, in § 6.1.3, that it "took one person-month" to develop. That sounds a lot more tractable than I would have guessed!

1 Like