I have been doing some work on shoring up some parts of Racket on Windows. There are a few moving parts to this work and I just want to give an update and get some feedback.
The racket-native-libs project is moving forward. Thanks to @mflatt being quite patient while I squashed some bugs, he was able to get the libraries built and copied using this project.
As part of this, there is a need to move Racket builds to use the ucrt libraries. ARM64 requires this anyway. This means there is no support for x86 cross-compiler tool and general support for 32-bit is going away from MSYS. The libraries have known will not fix bugs in thread local storage and other potential issues with multithread code.
For me, this means that the time for removing official support for 32-bit builds in Racket for Windows has come. The ecosystem around all of it is shutting down.
If parties really need 32-bit support on Windows, they can fork and build using MSVC and my project for the libraries. This is the best option for 32-bit support on Windows anyway.
I have a version of Racket (and related packages) that is built with MSVC that uses the libraries built with the racket-native-libs project on x64. The main changes were loading different libraries based on the value returned from (system-type 'so-find). There is some other general cleanup as well.
There is the issue of what Windows versions to support. ucrt can be loaded on Windows versions that are older than Windows 7 SP1. When it comes moving forward, it makes sense to support Windows Server 2008 as the baseline Win32 API. If it really becomes needed, moving up to Windows Server 2016 (the oldest version of Windows that is still in support) seems reasonable.
This is an issue. I have found and changed code that refers to NT4.0 in rktio. There is more work to be done there.
Does this mean that Windows builds of Racket will not be able to cross-compile for other platforms? Or does it mean that Racket on other platforms will not be able to cross-compile for Windows? Or both?
Maybe I'm not following, but I see a big gap between, on the one hand, deciding to no longer distribute “minimal Racket” Windows builds for 32-bit x86 and, on the other hand, requiring that, to make such builds, users fork and modify … something. If the modifications users would need to make are simple, can we include them in the mainline Racket source code, even if we need to stop distributing binaries for some reason?
For perspective, it sounds like this proposal would leave Windows on 32-bit x86 less supported than Mac OS on 32-bit PowerPC, which seems odd to me. Have I missed why we can't build Racket and native libraries against different C libraries for different architectures?
I would draw a distinction between upgrading native libraries for 32-bit x86 Windows and supporting Racket on x86 Windows. The way 32-bit PPC Mac OS works, for example, is that we don't try to upgrade native libraries; think of them as frozen in time, the same as the rest of the OS. Along similar lines, there's no need to support 32-bit x86 in new builds via racket-native-libs, but the old native libraries can stick around.
Yes, x86 refers to a 32-bit architecture in Windows terminology. The (system-type 'arch) result is 'i386 — which means something more like "i686", of course!
Sadly, for Windows on ARM, it does not. libiconv and libffi both use libtool which is a GNU tool to help managing shared libraries. The current version just plain breaks on clang-based cross complication. I looked at removing libtool from the builds. For libffi, it was major surgery. In addition, libffi is only officially supported on MSVC for Windows.
Again, the hope is that libtool gets it together with clang at some point and you can use the racket-native-libs project to cross-compile the needed Windows libraries on Linux. Also gcc based cross-compilers for aarch64 are on the roadmap, but it might be until 2027 until they are released.
Finally, it is not my intent to rebuild all the libraries for every release of Racket.
i686 builds work on racket-native-libs using MSVC. If Racket is cross-compiled for Windows using the llvm-mingw toolchain that uses ucrt for i686, x86_64, and aarch64, then my libraries will work in all three cases.
This excludes Windows XP or older because those OS versions can't support the ucrt runtime. I don't see any value in keeping msvcrt official builds (yes, you'd need i686 and x86_64) of Racket around. It would mean five builds and five sets of packages and having to change the logic on what DLL is loaded to detect if the OS was msvcrt only.
I would again suggest that Windows 7 SP1 or later being required for Racket on Windows is acceptable at this point.
There are two major C runtimes in Windows. msvcrt and ucrt. Using both as shared libraries in a executable is strongly discouraged and unsupported.
Racket on i686 and x86_64 targets msvcrt. Racket on aarch64 targets ucrt because Windows on ARM requires it.
There are updated libraries that will not cross-compile on a Linux host for a Windows (or mingw) target. This is due to libtool limitations in supporting clang cross-complication.
Therefore, racket-native-libs runs on Windows and uses the Microsoft compiler to build the updated windows libraries.
The updated libraries built with racket-native-libs target ucrt for all architectures. This is more consistent.
To use the updated libraries, Racket will need to build to target ucrt for all architectures. This should be supported by using the same cross-compiler toolchain (llvm-mingw) that is used to build Racket for Windows on ARM.
Racket is still cross-compiled on Linux for Windows.
The same codebase will be used to built all versions of Racket. There is no "Windows only" Racket, only specific native libraries needed by Racket on Windows.
Using ucrt means that Windows XP and older is no longer supported.
I think we're generally in sync here, but some clarifications, for what they're worth:
It really shouldn't matter which runtime the Racket executable itself uses. There should be no passing of C-allocated things like malloc results to be freed elsewhere or sometime like FILE* pointers from Racket. This is an explicit decision that goes way back. I'm pretty sure it still works out as intended, because I routinely work with /MT builds of Racket (the implications may be complicated, but it isn't msvcrt) while using native libraries that link to msvcrt.
It does matter that the native libraries in the sense of racket-native-libs link to a particular and specified runtime system, because those libraries almost certainly cooperate through the C library. And I think it's a fine idea to change x86_64 over to ucrt from msvcrt. At this point, the change seems more likely to solve than create problems for anyone who is building additional libraries.
The calculation seems different for 32-bit x86. It's nice that we could build 32-bit libraries using ucrt if needed, but I don't think we should bother. Just stick with the existing builds forever. Users of 32-bit x86 Racket on Windows (assuming there are some) don't have to get Cairo and Pango upgrades, anymore, just like Mac users for that architecture don't (except via MacPorts, but that's a different world).
For those interested in the Windows internals here, using the /MT flag statically links the ucrt (since VC++ 2015) library in an executable. For DLL files, the recommended default is to dynamically link to ucrt. Issues arise when both msvcrt and ucrt are used in DLLs loaded by a process and care is not taken to not marshall data properly. The recommended practice is use the same shared runtime in the executable and libraries.
Of course here are cases in which a statically linked runtime is needed (i.e. debugging and profiling) and in the case of Racket, this would have very little noticeable impact.
Completely agree this solves more issues at this juncture. All the MSYS environments that use msvcrt are marked as legacy and the ucrt64 environment is the preferred default.
No problem with that. Of course, if the toolchain dies out and we can't make 32 bit msvcrt builds of racket, I would assume that would be the end of line.
There are some details in how the Racket code that loads the libraries in packages like draw needs to be changed, as the new libraries have different names and additional dependencies and certain builds will use older libraries.
What I would propose is updating the define-runtime-lib macro to be more triplet-like. The most specific match wins and architecture always beats the so-find matches. Note, I am not using ffi-lib here as I expect this needs to be usable in ffi and ffi2. Also, this allows for error checks (you specified a dll for osx, that's wrong).
(define-runtime-lib fontconfig
[(linux) ; *-linux-*
(so "libfontconfig" "1")]
[(osx) ; *-osx-*
(dylib "libpng16.16")]
[(windows) ; *-windows-*
(dll "libfontconfig-2")]
; if the (system-type) is 'windows
; and (system-type 'so-find) is 'msvc
; use these libraries.
[(windows-msvc) ; *-windows-msvc-*
(dll "fontconfig-2")
; if the (system-type) is 'windows
; and (system-type 'arch) is 'i686
; use these libraries.
[(i686-windows) ; i686-windows-*
(dll "zlib1")
(dll "libfontconfig-1")])
and place this macro in a known package that can be imported where ever needed. I ran into a nasty bug where the older version of the macro was copied directly from the draw library to load libraries in the poppler package. I had to change the code in two places to support additional choices.
A lot of storage in racket consists of pointers.
32-bit pointers are half the size of 64-bit pointers.
this means that machines with limited memory are going
to be able to run Racket programs that are effectively
twice as big in 32-bit mode.