Well, I have been down quite the rabbit hole when it came to this project of updating the external libraries used by Racket and Racket packages. Learned quite a bit about CMake and vcpkg along the way. This work is mainly around Windows at the moment. I don't have access to a macOS environment and I am hoping to find somebody to see how this works on macOS.
Here's what I've learned.
Using VCPKG on Windows and MSVC works a treat. I have it making all the external libraries for x64, x86, and arm64. That's fifty-six DLLs in all. There are some hiccups that require rerunning cmake, but I am working on that. I have tested that things like openssl and sqlite work as expected.
I wanted to see if I could get a Linux based cross-compilation toolkit going. I tried. I really did. I was making custom overlays and patches. It's not there yet. ARM64 support is a nightmare and some libraries are a bit iffy for cross-compilation in general. Until the toolchains get better (and this could take a while), this path doesn't make sense.
I also looked at just using MSYS packages. This was all over the map as well, the main issue being that two key libraries in the MSYS packages are mutually linked and libffi can't handle that. Also, the x86 versions are officially unsupported as there are known thread-local storage bugs that wouldn't be fixed. Even the clang versions of MSYS has similar issues. And arm64 support is still in testing.
Where I need some input:
My proposal is to build all the external libraries for Windows on Windows. I will do some work and post the code on GitHub for people to try out. I need input on how to integrate this with the Racket release and CI environment down the road.
I want to start a discussion if it makes sense to use MSVC for the releases of Racket down the road. I know this brings up issues with the CI environment and I am very sensitive to that. The main issue is that the current racket build and tools use a completely ancient library versus using the UCRT.
I will start looking into if Racket itself could be built with a llvm-based toolchain that provides ucrt support on Linux of course.
I think we what we'd want is a set of Racket scripts to perform the build, where the output is in a form suitable to merge into GitHub - racket/libs: Racket libraries · GitHub. We can then upload (out of that repository, still as a manual step for now) new versions of packages for native libraries.
I expect that the DLLs that appear in each package will change, so we'd want to bump the version number on package names, such as "db-win32-arm64-2" and "draw-win32-i386-4".
I want to start a discussion if it makes sense to use MSVC for the releases of Racket down the road
I don't think this is going to be practical. Racket builds for all platforms are cross-compiled on Linux hosts via Docker.
That doesn't mean that we have to stick to MSVCRT for Windows, though. We could decide that Racket for Windows will switch to UCRT, and then we can set up new cross-compilation build-environment images for that choice, instead of the current MSVCRT choice. The current Docker images for building are defined by distro-build/docker/crosswin/Dockerfile at master · racket/distro-build · GitHub, so that's mostly the thing that would need to change.
Alternatively, I'm not sure there's any need for Racket itself to be linked against the same runtime library as expected of external libraries. Is there something that would prevent shifting all external libraries to UCRT while Racket remains MSVCRT?
That make sense. I can start on that. Also, we can go with a much slower release cadence on the libraries, like once-per-year or if a critical security issue hits.
This is all until the cross-compile tools catch up and we can move back to cross compiling via llvm for all the things.
[quote="mflatt, post:2, topic:4189"]
I don't think this is going to be practical. Racket builds for all platforms are cross-compiled on Linux hosts via Docker.
[quote]
I though so. Just wanted to confirm.
It is discouraged that a binary link to two different versions of the CRT, as issues can arise with things like I/O handles and the like. Also, there are some issues with using mingw64, msvcrt and winpthreads that using ucrt avoids (a set of nasty won't fix bugs with thread local storage).
I would suggest moving Racket to using the ucrt. ARM requires it anyway, and the UCRT is supported from Windows 7 SP1 on. It should be easy enough to do.
Also, it seems like this might be a good time to make a plan to update the docker files for Ubuntu 26.04 LTS and llvm-mingw 20240619 with LLVM 18.1.8 or consider LLVM 20 even.
I did a test build with the llvm-mingw toolchain for LLVM version 22 (link). Found a couple of minor issues due to C99 conformance. Patch was issued and accepted (thanks @mflat for the quick review). Compiling Racket against the UCRT had no issues. All that is required just installing the UCRT based toolchain from above.
I have created a GitHub project containing all the files I used to build these external libraries on Windows with MSVC. Link below.
Your consideration and feedback is requested. I need some guidance on how best organize the final library tarball to make it easy add the libraries to the needed raco packages.