Well, I have been down quite the rabbit hole when it came to this project of updating the external libraries used by Racket and Racket packages. Learned quite a bit about CMake and vcpkg along the way. This work is mainly around Windows at the moment. I don't have access to a macOS environment and I am hoping to find somebody to see how this works on macOS.
Here's what I've learned.
Using VCPKG on Windows and MSVC works a treat. I have it making all the external libraries for x64, x86, and arm64. That's fifty-six DLLs in all. There are some hiccups that require rerunning cmake, but I am working on that. I have tested that things like openssl and sqlite work as expected.
I wanted to see if I could get a Linux based cross-compilation toolkit going. I tried. I really did. I was making custom overlays and patches. It's not there yet. ARM64 support is a nightmare and some libraries are a bit iffy for cross-compilation in general. Until the toolchains get better (and this could take a while), this path doesn't make sense.
I also looked at just using MSYS packages. This was all over the map as well, the main issue being that two key libraries in the MSYS packages are mutually linked and libffi can't handle that. Also, the x86 versions are officially unsupported as there are known thread-local storage bugs that wouldn't be fixed. Even the clang versions of MSYS has similar issues. And arm64 support is still in testing.
Where I need some input:
My proposal is to build all the external libraries for Windows on Windows. I will do some work and post the code on GitHub for people to try out. I need input on how to integrate this with the Racket release and CI environment down the road.
I want to start a discussion if it makes sense to use MSVC for the releases of Racket down the road. I know this brings up issues with the CI environment and I am very sensitive to that. The main issue is that the current racket build and tools use a completely ancient library versus using the UCRT.
I will start looking into if Racket itself could be built with a llvm-based toolchain that provides ucrt support on Linux of course.
I think we what we'd want is a set of Racket scripts to perform the build, where the output is in a form suitable to merge into GitHub - racket/libs: Racket libraries · GitHub. We can then upload (out of that repository, still as a manual step for now) new versions of packages for native libraries.
I expect that the DLLs that appear in each package will change, so we'd want to bump the version number on package names, such as "db-win32-arm64-2" and "draw-win32-i386-4".
I want to start a discussion if it makes sense to use MSVC for the releases of Racket down the road
I don't think this is going to be practical. Racket builds for all platforms are cross-compiled on Linux hosts via Docker.
That doesn't mean that we have to stick to MSVCRT for Windows, though. We could decide that Racket for Windows will switch to UCRT, and then we can set up new cross-compilation build-environment images for that choice, instead of the current MSVCRT choice. The current Docker images for building are defined by distro-build/docker/crosswin/Dockerfile at master · racket/distro-build · GitHub, so that's mostly the thing that would need to change.
Alternatively, I'm not sure there's any need for Racket itself to be linked against the same runtime library as expected of external libraries. Is there something that would prevent shifting all external libraries to UCRT while Racket remains MSVCRT?
That make sense. I can start on that. Also, we can go with a much slower release cadence on the libraries, like once-per-year or if a critical security issue hits.
This is all until the cross-compile tools catch up and we can move back to cross compiling via llvm for all the things.
[quote="mflatt, post:2, topic:4189"]
I don't think this is going to be practical. Racket builds for all platforms are cross-compiled on Linux hosts via Docker.
[quote]
I though so. Just wanted to confirm.
It is discouraged that a binary link to two different versions of the CRT, as issues can arise with things like I/O handles and the like. Also, there are some issues with using mingw64, msvcrt and winpthreads that using ucrt avoids (a set of nasty won't fix bugs with thread local storage).
I would suggest moving Racket to using the ucrt. ARM requires it anyway, and the UCRT is supported from Windows 7 SP1 on. It should be easy enough to do.
Also, it seems like this might be a good time to make a plan to update the docker files for Ubuntu 26.04 LTS and llvm-mingw 20240619 with LLVM 18.1.8 or consider LLVM 20 even.
I did a test build with the llvm-mingw toolchain for LLVM version 22 (link). Found a couple of minor issues due to C99 conformance. Patch was issued and accepted (thanks @mflat for the quick review). Compiling Racket against the UCRT had no issues. All that is required just installing the UCRT based toolchain from above.
I have created a GitHub project containing all the files I used to build these external libraries on Windows with MSVC. Link below.
Your consideration and feedback is requested. I need some guidance on how best organize the final library tarball to make it easy add the libraries to the needed raco packages.
I tried following the directions here, but I got stuck at the build-deps.ps1 step. Is that script meant to be in the repo, perhaps along with copy-deps.ps1 , or am I missing something else?
My sincere apologies, the scripts were missing. I updated the repository with the scripts. Note the script names have changed. It should be Build-Racket-Libs.ps1 and Copy-Racket-Libs.ps1 now.
There are some packages that have external libraries that I could build and include. This included argon, libsass and libgit.
There are some packages that have external libraries I couldn't quite manage (ffmpeg was a real reach), so it is up to those package builders to create x86 and arm64 versions. Making x64 versions with ucrt might be necessary as well.
Also, I was thinking of using a different naming convention for the raco packages with newer libraries to avoid confusion:
[pkg]-windows-[arch]
An example being db-windows-arm64. This signals the packages use UCRT and may not work on Windows versions earlier than Win7 SP1.
Finally, I am looking at a change in how LoadLibrary is called so that the current directory of a library being loaded is temporarily added to the search path. If this works, we could replace code like:
(ffi-lib "zlib1.dll")
(ffi-lib "libiconv-2.dll")
(ffi-lib "libintl-9.dll")
; ... and so on...
(ffi-lib "libfontconfig-1.dll")])
with (ffi-lib "libfontconfig-1.dll") ; Loads all dependent libraries
Yes, this could make loading a library like gtk3 a one liner.
Is Build-Racket-Libs.ps1 still missing? (Minor: there's an extra b in bBuild-Racket-Libs.ps1 in the "Troubleshooting" section.)
I'd prefer not to change the package-name convention. Names like win32 and macosx are increasingly awkward, but they're less awkward than introducing more conventions. Bumping the version number at the end of the package name should be enough to cover the compatibility change.
Beware that changing the way LoadLibrary is called would not let us replace the enumeration of dependencies, because tools like raco exe need them spelled out. It's possible that changing the way LoadLibrary is called would be a good idea, but I worry about changing anything related to search paths. Search paths are not the route of all evil, but they pass through some pretty rough neighborhoods.
It was. The .gitignore file picked it up and missed that it was not in the commit. It seems I am quite rusty still.
Makes sense to me. I was just borrowing the convention from vcpkg and cmake.
The raco exe use case makes sense.
I agree that mucking around with search paths is problematic. My change is actually more restrictive than the standard LoadLibrary behavior and it seems to work just fine.
My change in rktio_dll.c first calls LoadLibraryExW with the LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR flag. This temporarily adds the current directory of the library to the search path and only looks there for resolving dependencies.
If that fails, it calls LoadLibraryExW with the LOAD_LIBRARY_SEARCH_SYSTEM32 flag that only looks in system directories for a DLL. So, this looks in fewer directories than LoadLibrary does.
Again, this also means that just loading the top level library via ffi-lib just does the right thing. No need to load the dependencies first in the proper order.
I am able to run Build-Racket-Libs.ps1, now, but I get an error about CMakeLists.txt being absent. Is that file missing, or am I doing something wrong?
This is my fault. I thought Github had all the files, but indeed, CMakeLists.txt was missing as well as a needed empty directory. I have pushed these changes.
Following on, I have made a number of changes in the project on GitHub and it is now in a much better state. There were a number of details missing and those have been fixed.
So, if you have access to a Windows machine and want to give this a try, please do so. I would appreciate it. Please note, this all done via the command line; no instructions on what to click where. I have added some experimental notes for Linux and MacOS if you are so inclined.
This requires a lot of patience. The gmp and mpfr libraries are absolute bears when it comes to compile times on Windows for some reason. Also, this is compiling gtk3 and all the related parts, so that takes time as well.
If you are wondering what is the point if you have to recompile all of this for one little change, the answer is simple. You don't. vcpkg caches build results and reuses them.
Again, the amount of work this saves versus tracking down every library for every platform can not be understated. This should reduce the burden of supporting Racket on all major operating system platforms.
As I finish this, the first build of libraries on Linux was successful.