FFI loading of raco-installed dynamic libraries

General
1

This might be better suited for "Internals", but being my first post to this forum... :wink:

One particular issue that exists is that of being able to ffi-lib dynamic libraries that have been installed by raco. This is especially frustrating on MacOS for reasons I'll go into.

The ffi-lib function appears to use dlopen to find libraries. This is great, except that if I have a system version of a library installed and use raco to install another version of the same library for my super-awesome-package, when I call ffi-lib (during development) it'll load the system-installed version, which can cause problems/conflicts.

On MacOS this can be extremely problematic for DrRacket, as dlopen uses environment variables for searching where it should load libraries: LD_LIBRARY_PATH, DYLD_LIBRARY_PATH, ... However, all of these variables are protected, and unset whenever a desktop application is launched. For example:

$ launchctl setenv FOO bar
$ launchctl setenv DYLD_LIBRARY_PATH /opt/homebrew/lib

// launch DrRacket

(getenv "FOO")   ; "bar"
(getenv "DYLD_LIBRARY_PATH")  ; #f

So I - as a MacOS user - have no control over where DrRacket searches for dynamic libraries I'm trying to load. It's 100% controlled by the program running via runtime search paths (i.e., @loader_path, @executable_path, and @rpath).

I can install dynamic libraries myself - for my packages - using raco, and they install to $HOME/Library/Racket/8.17/lib (on MacOS, and I assume another similarly suitable place on Windows/Linux), but racket/DrRacket never looks for them there at any time other than compilation and adding them to the final, distributed program.

I think it would phenominally awesome if Racket (and DrRacket) first attempted to load dynamic libraries from the raco installed set of dynamic libraries, and then - if not found - could fall-back to the system paths.

I believe this would fix both DrRacket on MacOS and the global issue of loading dynamic libraries from the system before package-installed versions.

And if there's currently an existing workaround (or setting) for this that I haven't found yet, please let me know. :slight_smile:

2

It may help to turn on logging for ffi-lib to see how it's searching. In DrRacket, show the logging panel ("View" -> "Show Log") and include debug@ffi-lib in the configuration box at the top of the panel.

It should search in ~/Library/Racket/8.17/lib first. A common problem is that a shared library is found in the filesystem, but it doesn't load successfully for some reason, and so the search moves on. (Maybe it doesn't load because it has a dependency that isn't found, for example.) Sometimes, enough of an error message shows up in the log to hint at what has gone wrong. It's possible for the search path to become misconfigured, but that's less common. You can also use an absolute path to a library to make sure of the that that is tried.

Another possible confusion is that once DrRacket has picked a shared library to load, the library will not be loaded again until DrRacket is restarted. That's because DrRacket doesn't create a new process to run your program, and it doesn't unload shared libraries within its own process.

1 Like
3

That does help (and is fantastic news!), as I can see that it does search the raco install library, does find the file there, but then continues on to the Application folder, then using the os search paths:

ffi-lib: failed for (ffi-lib "libcsfml-window.2.6.1.dylib" ""), tried: 
  #<path:/Users/jeff/Library/Racket/8.17/lib/libcsfml-window.2.6.1.dylib> (exists)
  #<path:/Users/jeff/Library/Racket/8.17/lib/libcsfml-window.2.6.1.dylib> (exists)
  #<path:/Applications/Racket v8.17/lib/libcsfml-window.2.6.1.dylib> (no such file)
  #<path:/Applications/Racket v8.17/lib/libcsfml-window.2.6.1.dylib> (no such file)
  "libcsfml-window.2.6.1.dylib" (using OS library search path)
  "libcsfml-window.2.6.1.dylib" (using OS library search path)
  #<path:/Users/jeff/Developer/Racket/racket-csfml/libcsfml-window.2.6.1.dylib> (no such file)
  #<path:/Users/jeff/Developer/Racket/racket-csfml/libcsfml-window.2.6.1.dylib> (no such file)

I'm assuming here that it attempted to load the library and it failed for a reason it didn't provide in the log (e.g., wrong arch)?

4

Yes. In this situation, I usually try loading the library using the absolute path like "/Users/jeff/Library/Racket/8.17/lib/libcsfml-window.2.6.1.dylib", and that can provide more of an error message.

We should change the internals so that ffi-lib can log an error message when a step in its search path fails. It's also questionable to continue searching when a load fails — but that's ultimately a consequence of dlopen saying why it failed only with a string-based message, instead of something easy to check programmatically.

5

For completeness, loading the library directly from the raco path yielded the error:

Reason: no LC_RPATH's found

And I was able to get this rectified via other means. Thanks so much for the information. And I'm glad what I thought would be a good solution was already there. :slight_smile:

1 Like