Distcc adventures: Distributed cross-compiling with macOS and Windows/Linux
This post was originally written on April 20, and it uses LLVM 11.1
Recently I've had a sudden urge to do some distributed compilation to speed up the compilation times for my compact Vulkan engine (which is written in C++). It's not a big codebase by any means, but mostly being on an 8 year old MacBook (it's comfy), I thought it'd be nice to leverage the computational power of my Windows PC, which usually just plays some jazz or lo-fi streams in the background.
However, this innocent adventure turned into a slightly arduous journey. So I decided to turn this experience into a quick guide.
What we're going to do is to cross-compile and produce a macOS binary with distributed compilation on both macOS and Windows nodes (which is actually Linux via WSL).
Here's a condensed version of all the steps. Please note that, I'm on an x86 MacBook, but you should be able to replicate this for M1 as well.
WSL: The magic ingredient
We'll need WSL on Windows to have access to a Linux environment. This is of course not needed if you're running Linux natively.
I've gone with WSL1, since WSL2 seems to suffer from major memory issues.
Ubuntu works nicely, but others should work as well.
Upgrade Ubuntu to the non-LTS version
WSL Ubuntu is LTS by default. We'll turn that into a non-LTS version to have access to the latest packages. (Ubuntu 20.04 focal ⮕ 20.10 groovy)
- Do a full system upgrade:
sudo apt update -y && sudo apt upgrade -y
- Use the snippet below, or edit
/etc/apt/sources.list, search & replace
groovyin order to use the latest repositories. (Names will be different depending on the Ubuntu versions in the future)
sudo sed -i 's/focal/groovy/g' /etc/apt/sources.list
- Final full system upgrade:
sudo apt update -y && sudo apt upgrade -y
We need to install the same version of clang on all machines. This can be a bit of a manual work if package systems don't provide the same version.
I tend to use a manually installed clang on macOS. This is primarily because Apple's Clang lacks the leak sanitizer. Also, we need a specific version of llvm/clang in order to use it with distcc.
We can install it with the help of homebrew:
brew install llvm
Which version we install is important. At the time of this post,
11.1 is what is available on homebrew. So we'll stick with that.
As the brew package info mentions, we'll need to add llvm's bin folder to our path.
Install Xcode CommandLineTools
We'll be using the linker from
CommandLineTools. This is because llvm's own linker on macOS does not seem to be fully functional yet. Therefore we're stuck with the latest and greatest linker Apple has ever provided.
If you have Xcode installed, you can install the command line tools from there.
If you don't have Xcode installed, you can grab
CommandLineTools installer from Apple's developer downloads page (You'll need to login). Not having to install Xcode should save you a fair amount of disk space.
Linux setup (WSL)
Turns out, even the non-LTS Ubuntu does not have the clang version we need (11.1), so we'll need to install it manually.
We can simply grab one of the precompiled releases, and unpack it in our home folder.
curl -OL https://github.com/llvm/llvm-project/releases/download/llvmorg-11.1.0/clang+llvm-11.1.0-x86_64-linux-gnu-ubuntu-20.10.tar.xz
tar vfxJ clang+llvm-11.1.0-x86_64-linux-gnu-ubuntu-20.10.tar.xz
For convenience, we can either rename the folder to
llvm or create a symlink to it.
mv clang+llvm-11.1.0-x86_64-linux-gnu-ubuntu-20.10 llvm
We'll need to update our path to include
llvm/bin folder and we're nearly set.
Grab crt from the gcc package
Nearly, because llvm/clang on Linux is actually using crt from gcc by default. So we'll need to install the corresponding package.
sudo apt install libgcc-10-dev
This should give us the goods to proceed.
We can finally install distcc.
distcc on Linux (our only node in our compile farm)
I opted to go with the Ubuntu package, though it's also possible to compile manually. (I've tried this, but distcc had a few hard-coded paths in its symlinking script, and the uninstall target had issues at the time I tried)
sudo apt install distcc
We need to tell distcc which compilers are supported. This can be done automatically by firing up the script that is provided by the package:
This should create necessary symlinks under
/usr/lib/distcc. It should look like this:
➜ /usr/lib/distcc ls -l
lrwxrwxrwx 1 root root 16 Apr 1 20:04 c++ -> ../../bin/distcc*
lrwxrwxrwx 1 root root 16 Apr 1 20:04 c89 -> ../../bin/distcc*
lrwxrwxrwx 1 root root 16 Apr 1 20:04 c99 -> ../../bin/distcc*
lrwxrwxrwx 1 root root 16 Apr 1 20:04 cc -> ../../bin/distcc*
lrwxrwxrwx 1 root root 16 Apr 1 20:04 clang -> ../../bin/distcc*
lrwxrwxrwx 1 root root 16 Apr 1 20:04 clang++ -> ../../bin/distcc*
lrwxrwxrwx 1 root root 16 Apr 1 20:04 clang++-11 -> ../../bin/distcc*
lrwxrwxrwx 1 root root 16 Apr 1 20:04 clang-11 -> ../../bin/distcc*
However, the script didn't seem to work in my case. So we can simply create them manually. Note that, these all point to the
distcc binary, but I think distcc only checks the presence of the symlink.
Launch the daemons!
Now it is time for a test run, let's launch distcc:
distccd --no-detach --daemon --allow 127.0.0.1 --allow 192.168.1.0/24 --listen 192.168.1.42 --nice 10 --log-stderr
Now we're only allowing connections from our localhost and LAN, with a mask. We're also only listening on the LAN interface, where
192.168.1.42 is the IP of the server on our LAN.
nice parameter deprioritizes the processes, so it doesn't stall our jazz stream.
--log-stderr will print the log to stderr, we can additionally pass
--verbose here to debug if something goes wrong.
If you have more computers, repeat this step to install distcc + llvm/clang so they're all available as nodes.
Open them firewalls
Don't forget to adjust your firewall(s) so your computers can communicate with each other. distcc uses tcp port 3632 by default. (It is also possible to use an ssh tunnel)
distcc on macOS (our main machine)
brew install distcc
Using distcc as a compiler
Now, we just need to use
distcc to compile on our main machine.
First, we need to define an environment variable for
distcc to quickly find out which hosts are available as distcc nodes.
export DISTCC_HOSTS='localhost/4 bahamut/10'
Bahamut is the name of my PC, but you can also put IP addresses. The number after the slash is the number of cores distcc should utilize on the target node. Generally it's nice to increase this number by one or two, so we can fill out all the cores during stalls.
Note that it's possible to prevent compilation on localhost entirely. Simply remove
DISTCC_HOSTS and set:
This prevents falling back to localhost if a remote compilation fails. This should allow you to compile everything on remote nodes at all times.
We need to set our compiler to
distcc, and tell distcc to use
clang (and avoid Apple's binaries).
The easiest way of doing this is to set
CXX environment variables, however you can also make adjustments in your build system.
export CC='distcc clang'
export CXX='distcc clang++'
Here distcc acts as a compiler driver.
The target triplet
Now, in order for cross-compiling to work, we need to adjust our build system to always set
-target parameter. This tells clang to produce code for the right architecture and feature set.
You can find the target triplet on the host computer from clang:
➜ clang --version
clang version 11.1.0
Thread model: posix
In this case,
x86_64-apple-darwin20.3.0 is the triplet I need to use in order to produce binaries for my macOS.
When using CMake, we can simply use something like this:
However, we can automate this completely by getting the triplet with
So instead of the manual
SET statement above, we can set
TARGET with the output of this command:
execute_process(COMMAND clang -dumpmachine OUTPUT_VARIABLE TARGET OUTPUT_STRIP_TRAILING_WHITESPACE)
Accommodate for the number of (virtual) cores
Make sure to pass an adequate number of cores to your build system, so it will parallelize the compilation and accommodate for the number of cores in our compile farm.
It is possible to derive this number from distcc itself:
For example, if you're using Make on Bash, you can use:
make -j`distcc -j`
If you're using CMake on Fish:
cmake --build build --config Debug -j (distcc -j)
If you want some extra oomph:
Make / Bash:
make -j$((`distcc -j` + 4))
CMake / Fish:
cmake --build build --config Debug -j (math (distcc -j) + 4)
Now, we can
finally simply compile stuff with our regular build systems, and it should distribute it to the remote nodes.
We're not using distcc's
pump mode, as I didn't find it necessary in my setup. If you have a massive compile farm and huge codebases, you might want to consider using it.
Debugging & monitoring
If something goes wrong, it's possible to launch your build system with
DISTCC_VERBOSE=1 cmake --build ...
You can also monitor distcc using its command line tool:
1 is the number of seconds between updates, adjust if necessary.
There is also a GUI version of this tool, written for gnome. If your system supports it, you might want to check it out.
But wait, there's more
ccache, but we can take a look at that in another blog post.
That's all, for now!
Questions or comments? Hit me up on Twitter.