Many times, during these last months, I thought to keep updated my blog writing a new post. Unfortunately, for one or another reason I always found an excuse to not do so. Well, I think that time is over because finally I found something useful and worthy the time spent time on the writing.
– That is OK but … what are you talking about?.
– Be patient Pablo, if you didn’t skip the headline of the post you already know about what I’m talking, probably :-).
Yes, I’m talking about how to setup a MacPro computer into a icecc cluster based on Linux hosts to take advantage of those to get more CPU power to build heavy software projects, like Chromium, faster. The idea besides this is to distribute all the computational work over Linux nodes (fairly cheaper than any Mac) requested for cross-compiling tasks from the Mac host.
I’ve been working as a sysadmin at Igalia for the last couple of years. One of my duties here is to support and improve the developers building infrastructures. Recently we’ve faced long building times for heavy software projects like, for instance, Chromium. In this context, one of the main issues that I had to solve is how to build Chromium for MacOS in a reasonable time and avoiding to spend a lot of money in expensive bleeding edge Apple’s hardware to get CPU power.
This is what this post is about. This is an explanation about how to configure a Mac Pro to use a Linux based icecc cluster to boost the building times using cross-compilation. For simplicity, the explanation is focused in the singular case of just one single Linux host as icecc node and just one MacOS host requesting for compiling tasks but, in any case, you can extrapolate the instructions provided here to have many nodes as you need.
So let’s go with the the explanation but, first of all, a summary for those who want to go directly to the minimal and essential information …
TL;DR
On the Linux host:
# Configure the iceccd
$ sudo apt install icecc
$ sudo systemctl enable icecc-scheduler
$ edit /etc/icecc/icecc.conf
ICECC_MAX_JOBS="32"
ICECC_ALLOW_REMOTE="yes"
ICECC_SCHEDULER_HOST="192.168.1.10"
$ sudo systemctl restart icecc
# Generate the clang cross-compiling toolchain
$ sudo apt install build-essential icecc
$ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git ~/depot_tools
$ export PATH=$PATH:~/depot_tools
$ git clone https://github.com/psaavedra/chromium_clang_darwin_on_linux ~/chromium_clang_darwin_on_linux
$ cd ~/chromium_clang_darwin_on_linux
$ export CLANG_REVISION=332838 # or CLANG_REVISION=$(./get-chromium-clang-revision)
$ ./icecc-create-darwin-env
# copy the clang_darwin_on_linux_332838.tar.gz to your MacOS host
On the Mac:
# Configure the iceccd
$ git clone https://github.com/darktears/icecream-mac.git ~/icecream-mac/
$ sudo ~/icecream-mac/install.sh 192.168.1.10
$ launchctl load /Library/LaunchDaemons/org.icecream.iceccd.plist
$ launchctl start /Library/LaunchDaemons/org.icecream.iceccd.plist
# Set the ICECC env vars
$ export ICECC_CLANG_REMOTE_CPP=1
$ export ICECC_VERSION=x86_64:~/clang_darwin_on_linux_332838.tar.gz
$ export PATH=~/icecream-mac/bin/icecc/:$PATH
# Get the depot_tools
$ cd ~
$ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
$ export PATH=$PATH:~/depot_tools
# Download and build the Chromium sources
$ cd chromium && fetch chromium && cd src
$ gn gen out/Default --args='cc_wrapper="icecc" \
treat_warnings_as_errors=false \
clang_use_chrome_plugins=false \
use_debug_fission=false \
linux_use_bundled_binutils=false \
use_lld=false \
use_jumbo_build=true'
$ ninja -j 32 -C out/Default chrome
… and now the detailed explanation
Installation and setup of icecream on Linux hosts
The installation of icecream on a Debian based Linux host is pretty simple. The latest version (1.1) for icecc is available in Debian testing and sid for a while so everything that you must to do is install it from the APT repositories. For case of stretch, there is a backport available in the apt.igalia.com repository publically available:
sudo apt install icecc
The second important part of a icecc cluster is the icecc-scheduler. This daemon is in charge to route the requests from the icecc nodes which requiring available CPUs for compiling to the nodes of the icecc cluster allowed to run remote build jobs.
In this setup we will activate the scheduler in the Linux node (192.168.1.10). The key here is that only one scheduler should be up at the same time in the same network to avoid errors in the cluster.
sudo systemctl enable icecc-scheduler
Once the scheduler is configured and up, it is time to add icecc hosts to the cluster. We will start adding the Linux hosts following this idea:
We will need to restart the service to apply those changes:
sudo systemctl restart icecc
Installing and setup of icecream on MacOS hosts
The next step is to install and configure the icecc service on our Mac. The easy way to get icecc available on Mac is icecream-mac project from darktears. We will do the installation assuming the following facts:
- The local user account in Mac is psaavedra
- The IP of the icecc scheduler is 192.168.1.10
- The Mac is not allowed to accept remote jobs
- We don’t want run use the Mac as worker.
To get the icecream-mac software we will make a git-clone of the project on Github:
git clone https://github.com/darktears/icecream-mac.git /Users/psaavedra/icecream-mac/
sudo /Users/psaavedra/icecream-mac/install.sh 192.168.1.10
We will edit a bit the /Library/LaunchDaemons/org.icecream.iceccd.plist daemon definition as follows:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>org.icecream.iceccd</string>
<key>ProgramArguments</key>
<array>
<string>/Users/psaavedra/icecream-mac/bin/icecc/iceccd</string>
<string>-s</string>
<string>192.168.1.10</string>
<string>-m</string>
<string>2</string>
<string>--no-remote</string>
</array>
<key>KeepAlive</key>
<true/>
<key>UserName</key>
<string>root</string>
</dict>
</plist>
Note that we are setting 2 workers in the Mac. Those workers are needed to execute threads in the host client host for things like linking … We will reload the service with this configuration:
launchctl load /Library/LaunchDaemons/org.icecream.iceccd.plist
launchctl start /Library/LaunchDaemons/org.icecream.iceccd.plist
Getting the cross-compilation toolchain for the icecream-mac
We already have the icecc cluster configured but, before to start to build Chromium on MacOS using icecc, there is still something before to do. We still need a cross-compiled clang for Darwin on Linux and, to avoid incompatibilities between versions, we need a clang based on the very same version that your Chromium code to be compiled.
You can check and get the cross-compilation clang revision that you need as follows:
cd src
CLANG_REVISION=$(cat tools/clang/scripts/update.py | grep CLANG_REVISION | head -n 1 | cut -d "'" -f 2)
echo $CLANG_REVISION
332838
In order to simplify this step. I made some scripts which make it easy the generation of this clang cross-compiled toolchain. On a Linux host:
- Install build depends:
sudo apt install build-essential icecc
- Get the Chromium project depot tools
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git ~/depot_tools
export PATH=$PATH:~/depot_tools
- Download the psaavedra’s scripts (yes, my scripts):
git clone https://github.com/psaavedra/chromium_clang_darwin_on_linux ~/chromium_clang_darwin_on_linux
cd ~/chromium_clang_darwin_on_linux
- You can use the get-chromium-clang-revision script to get the latest clang revision using in Chromium master:
./get-chromium-clang-revision
- and then, to build the cross-compiled toolchain:
./icecc-create-darwin-env
; this script encapsulates the download, configure and build of the clang software.
- A clang_darwin_on_linux_999999.tar.gz file will be generated.
Setup the icecc environment variables
Once you have the /Users/psaavedra/clang_darwin_on_linux_332838.tar.gz generated in your MacOS. You are ready to set the icecc environments variables.
export ICECC_CLANG_REMOTE_CPP=1
export ICECC_VERSION=x86_64:/Users/psaavedra/clang_darwin_on_linux_332838.tar.gz
The first variable enables the usage of the remote clang for C++. The second one establish toolchain to use by the x86_64 (Linux nodes) to build the code sent from the Mac.
Finally, remember to add the icecc binaries to the $PATH:
export PATH=/Users/psaavedra/icecream-mac/bin/icecc/:$PATH
You can check and get the cross-compiled clang revision that you need as follows:
cd src
CLANG_REVISION=$(cat tools/clang/scripts/update.py | grep CLANG_REVISION | head -n 1 | cut -d "'" -f 2)
echo $CLANG_REVISION
332838
… and building Chromium, at last
Reached this point, it’s time to build a Chromium using the icecc cluster and the cross-compiled clang toolchain previously created. These steps follows the official Chromium build procedure and only adapted to setup the icecc wrapper.
Ensure depot_tools is the path:
cd ~git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
export PATH=$PATH:~/depot_tools
ninja --version
# 1.8.2
Get the code:
git config --global core.precomposeUnicode truemkdir chromium
cd chromium
fetch chromium
Configure the build:
cd src
gn gen out/Default --args='cc_wrapper="icecc" treat_warnings_as_errors=false clang_use_chrome_plugins=false linux_use_bundled_binutils=false use_jumbo_build=true'
# or with ccache
export CCACHE_PREFIX=icecc
gn gen out/Default --args='cc_wrapper="ccache" treat_warnings_as_errors=false clang_use_chrome_plugins=false linux_use_bundled_binutils=false use_jumbo_build=true'
And build, at last:
ninja -j 32 -C out/Default chrome
icemon allows you to graphically monitoring the icecc cluster. Run it in remote from your Linux host if you don’t want install it in the MacOS:
ssh -X user@yourlinuxbox icemon
; with icemon you should see how each build task is distributed across the icecc cluster.
503: Service Unavailable 