# Parallel compilation support? **URL:** \ **Category:** Distro Packagers\ **Created:** [November 30, 2023, 8:03pm UTC](https://racket.discourse.group/t/parallel-compilation-support/2557 "2023-11-30T20:03:22Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![zyga](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/zyga/32/585_2.png) [@zyga](https://racket.discourse.group/u/zyga)\ **Post date:** [November 30, 2023, 8:03pm UTC](https://racket.discourse.group/t/parallel-compilation-support/2557/1 "2023-11-30T20:03:22Z")
I've realized that while building racket, a lot of the action happens on a single core. Invoking `make -j16` at the top level, I see two things: - zuo apparently running things sequentially (please correct me if I'm wrong) - some nested libraries are being built sequentially, even though they are using make ```scheme :: make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. :: make[1]: Entering directory '/root/parts/racket/build/src/cs/c/ChezScheme/ta6le/lz4/lib' ``` As core counts raise, this makes a lot of the build machine idle. Is there a way to address this easily? ---
**Author:** ![samth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/samth/32/3_2.png) [@samth](https://racket.discourse.group/u/samth)\ **Post date:** [November 30, 2023, 8:55pm UTC](https://racket.discourse.group/t/parallel-compilation-support/2557/2 "2023-11-30T20:55:09Z")
zuo will run multiple jobs in parallel, and it will communicate the jobserver to recursive make calls. You can see a transcript here where `-j 18` is automatically communicated to the lower levels, including to `raco setup`: [DrDr / R65264 / pkg-src / build / make](http://drdr.racket-lang.org/65264/pkg-src/build/make) ---
**Author:** ![zyga](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/zyga/32/585_2.png) [@zyga](https://racket.discourse.group/u/zyga)\ **Post date:** [November 30, 2023, 9:12pm UTC](https://racket.discourse.group/t/parallel-compilation-support/2557/3 "2023-11-30T21:12:35Z")
Thanks for responding. I think that either my setup has something special that breaks this or my expectations are off: I see this process tree: ```scheme │ │ │ └─make -j16 │ │ │ └─sh -c bin/zuo . all MAKE="make" │ │ │ └─zuo . all MAKE=make │ │ │ └─racketcs -X /root/parts/racket/build/collects -G /root/parts/racket/build/src/cs/c/compiled/etc -R /root/parts/racket/build/src/cs/c/compiled -N raco -l- setup --no-user --no-launcher --no-foreign-libs --no-install --no-post-install --no-pkg-deps │ │ │ └─36*[{racketcs}] ``` With the build process producing: ```scheme :: raco setup: 7 making: /gui-doc/mrlib/scribblings/hierlist :: raco setup: 7 making: /gui-lib/mrlib :: raco setup: 7 making: /gui-lib/mrlib/hierlist :: raco setup: 7 making: /gui-lib/mrlib/private :: raco setup: 7 making: /gui-lib/mrlib/private/aligned-pasteboard :: raco setup: 7 making: /gui-lib/scheme :: raco setup: 7 making: /gui-lib/scheme/gui :: raco setup: 7 making: /gui-lib/scheme/gui/lang :: raco setup: 7 making: /gui-lib/scribble :: raco setup: 7 making: /gui-lib/scribble/private :: raco setup: 7 making: /gui-pkg-manager-lib/pkg :: raco setup: 7 making: /gui-pkg-manager-lib/pkg/gui :: raco setup: 7 making: /gui-pkg-manager-lib/pkg/gui/private :: raco setup: 7 making: /htdp-doc/graphics :: raco setup: 7 making: /htdp-doc/graphics/scribblings ``` With the build machine being largely idle. Is that expected? ---
**Author:** ![samth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/samth/32/3_2.png) [@samth](https://racket.discourse.group/u/samth)\ **Post date:** [November 30, 2023, 9:23pm UTC](https://racket.discourse.group/t/parallel-compilation-support/2557/4 "2023-11-30T21:23:11Z")
Definitely that process (running `raco setup`) will be the only one running for a large portion of the time, but it will use multiple cores. However, it looks like the jobserver isn't propagating the `-j` option to `raco setup` properly and thus it isn't using 16 cores (7 is the process number so it's at least using 7). What is `make` on your system, and what version of Racket are you building? ---
**Author:** ![zyga](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/zyga/32/585_2.png) [@zyga](https://racket.discourse.group/u/zyga)\ **Post date:** [November 30, 2023, 9:23pm UTC](https://racket.discourse.group/t/parallel-compilation-support/2557/5 "2023-11-30T21:23:28Z")
Looking some more, there are multiple stages during the build, this one looks particularly sequential: ```scheme │ │ │ └─scheme -q cs/c/ChezScheme/xc-ta6le/s/cmacros.so cs/c/ChezScheme/xc-ta6le/s/priminfo.so cs/c/ChezScheme/xc-ta6le/s/primvars.so cs/c/ChezScheme/xc-ta6le/s/env.so cs/c/ChezScheme/xc-ta6le/s/setup.so cs/c/ChezScheme/xc-ta6le/s/xpatch ``` This lights up one core, hinting at internal sequentially. ---
**Author:** ![zyga](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/zyga/32/585_2.png) [@zyga](https://racket.discourse.group/u/zyga)\ **Post date:** [November 30, 2023, 9:24pm UTC](https://racket.discourse.group/t/parallel-compilation-support/2557/6 "2023-11-30T21:24:58Z")
I'm building racket 8.11.1 with make 4.3 as in Ubuntu 22.04 ---
**Author:** ![samth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/samth/32/3_2.png) [@samth](https://racket.discourse.group/u/samth)\ **Post date:** [November 30, 2023, 9:25pm UTC](https://racket.discourse.group/t/parallel-compilation-support/2557/7 "2023-11-30T21:25:57Z")
As in, the "source" build from here: [Racket: Racket 8.11.1](https://download.racket-lang.org/releases/8.11.1/) ? ---
**Author:** ![zyga](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/zyga/32/585_2.png) [@zyga](https://racket.discourse.group/u/zyga)\ **Post date:** [November 30, 2023, 9:26pm UTC](https://racket.discourse.group/t/parallel-compilation-support/2557/8 "2023-11-30T21:26:28Z")
I'm using this tarball: [https://download.racket-lang.org/releases/8.11.1/installers/racket-8.11.1-src.tgz](https://download.racket-lang.org/releases/8.11.1/installers/racket-8.11.1-src.tgz) ---
**Author:** ![samth](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/samth/32/3_2.png) [@samth](https://racket.discourse.group/u/samth)\ **Post date:** [November 30, 2023, 10:09pm UTC](https://racket.discourse.group/t/parallel-compilation-support/2557/9 "2023-11-30T22:09:44Z")
Ok, I see this same behavior with the "source" tarball. When I build from the git sources at [GitHub - racket/racket: The Racket repository](https://github.com/racket/racket) using the top-level `make` target then I get appropriate parallelism, but if I directly go to `racket/src` and run `configure` and then `make -j 8` I again get the jobserver warning and no parallelism. So this seems like a bug somewhere in the build system. Probably @mflatt is the right person to look into this. ---
**Author:** ![zyga](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/zyga/32/585_2.png) [@zyga](https://racket.discourse.group/u/zyga)\ **Post date:** [November 30, 2023, 10:10pm UTC](https://racket.discourse.group/t/parallel-compilation-support/2557/10 "2023-11-30T22:10:40Z")
Thanks for debugging this. I'm sure that with enough eyeballs this can be addressed. When there's a patch, even before the full release, I'd be happy to cherry-pick to make iteration faster. ---
**Author:** ![mflatt](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/mflatt/32/6_2.png) [@mflatt](https://racket.discourse.group/u/mflatt)\ **Post date:** [December 1, 2023, 2:55am UTC](https://racket.discourse.group/t/parallel-compilation-support/2557/11 "2023-12-01T02:55:21Z")
I've just pushed a change to the makefile stub for `racket/src`, adding the `+`s that are needed to propagate `-j` and jobserver pipes for some versions of GNU Make (especially newer ones). That depends on some post-8.11 fixes already in place at the Zuo level. Those changes don't address the `scheme -q cs/c/ChezScheme/xc-ta6le/s/cmacros.so ...` step in building Chez Scheme. The zuofile at that level has two targets: one builds files independently so that they can happen in parallel, and one builds in a single `scheme` process. The latter is mostly used — imitating historical behavior, which was probably chosen because compilation is so fast when the compiler itself is compiled to native code. When the compiler is running slowly through pb interpretation in the bootstrap phase, though, separate compile processes would probably make more sense. I haven't yet tried changing that.