A good compromise: Cross-compiling with distcc

Cross-compiling for the Raspberry Pi using distcc

[ Update: If you’re looking to build a minimal rootfs, want to cross-compile in a more traditional way, or want to build using QEMU, then take a look at some of my other posts. Thanks! ]

If you’re new to Linux, or to the Raspberry Pi, you might feel like we’re pushing the leading edge. And we are in several ways; the Pi is an absolute miracle of technology, but mostly due to its size and price; it’s not very fast compared to modern computers.

Years ago, building large programs like the Linux kernel could take hours on what was then a modern workstation. So, undoubtedly while waiting on a compile, the folks behind Samba were staring at one machine running as hard as it could while dozens of others in the office sat idle. They wondered if they could find a way to easily distribute the build tasks to all the machines in the office, and so they wrote something called distcc.

I won’t go into all the details here, but basically distcc intercepts calls to the regular compiler and sends the work to other machines over the network. The other, slave machines do the actual compilation and send the object files or error messages back to the master computer, who then links them together.

Do you see where I’m going with this?

We’re going to use distcc on the Pi. Instead of building the files locally, the Pi is going to send them over the network to a fast computer which is running an ARM cross-compiler. The network computer will send back ARM objects to the Pi.

The beautiful thing about this is that as far as the package that’s being built can tell, this is a local compile. There’s some initial setup, but on a per-package basis, there are no hoops to jump through in order to leverage the faster computer.

Setting up the slave computer

I did the following on my large, modern Linux machine:

Setup the cross-compiler

I grabbed the pre-built cross compilers from the RPi Foundation’s repository https://github.com/raspberrypi/tools. Specifically, I installed the compilers in /opt/cross so that they looked like this:

$ ls -l /opt/cross/arm-bcm2708/
drwxr-xr-x 7 root root 4096 Oct  4 11:05 arm-bcm2708hardfp-linux-gnueabi
drwxr-xr-x 7 root root 4096 Oct  4 11:05 arm-bcm2708-linux-gnueabi
drwxr-xr-x 7 root root 4096 Oct  4 11:05 gcc-linaro-arm-linux-gnueabihf-raspbian

You can also get an RPM of the compilers that will place them in /opt/cross from https://github.com/downloads/midnightyell/RPi-LTIB/raspberrypi-tools-9c3d7b6-1.i386.rpm. I installed this rpm on my Ubuntu system using the command:

sudo rpm -i --ignorearch raspberrypi-tools-9c3d7b6-1.i386.rpm

Note that this is the same file, installed in the same location as used for LTIB. If you’ve already installed LTIB, you already have these compilers installed in the correct place.

Install distcc

sudo apt-get install distcc

Then edit /etc/default/distcc so that it looks like this:

STARTDISTCC="true"
ALLOWEDNETS="0.0.0.0/0"
LISTENER=""
NICE="0"
JOBS=""
ZEROCONF="true"
PATH=/opt/cross/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/arm-linux-gnueabihf/bin/:\
/opt/cross/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/libexec/gcc/arm-linux-gnueabihf/4.7.2:\
${PATH}

You may wish to adjust the ALLOWEDNETS and LISTENER parameters to suit your network, but these defaults should work. I’ve also adjusted the JOBS parameter to be a largish number (50) so that the Pi will send work as fast as it can generate it to my desktop machine.

Now start the distcc daemons

sudo /etc/init.d/distcc restart

Setting up the Raspberry Pi

I started with the latest Raspbian SD card image, did the usual setup including

sudo apt-get update
sudo apt-get upgrade

Followed by

sudo apt-get install distcc

Make sure that when “gcc” is called, it actually calls distcc:

export PATH=/usr/lib/distcc:${PATH}

Tell the RPi what hosts to use

Edit ~/.distcc/hosts to resemble the following:

192.168.1.100
--localslots=1
--randomize

Where the ipaddress is that of your slave computer; .100 in my example. If you have more than one slave, add them to this file, one per line.

Let’s compile MAME!

wget http://prdownloads.sourceforge.net/advancemame/advancemame-0.106.1.tar.gz
tar xvzf advancemame-0.106.1.tar.gz
cd advancemame-0.106.1
./configure
make -j8

Conclusions

AdvanceMAME took between 12 and 15 minutes to build in this way on my machine, and I’m pretty sure the Pi was the limiting factor. As a comparison, it took 42 minutes to build in QEMU, and under 1 minute to cross-compile on this same machine. One of these days, I’ll time the build on the Pi, but I suspect it’s in the 2-hour range. [ Update: 55 minutes with the Pi in turbo mode at 1GHz; this really makes QEMU (at 42 minutes) look bad! ]

Overall, I like this technique. Once it’s set up, it should work automatically. I didn’t have to fight any of the Makefiles in order to get things to build, and it worked much, much faster than a native or an emulated build. And — once I solve the Pi bottleneck — I can keep throwing spare machines at this; any Linux machine sitting idle is a candidate to be turned into a distcc server. [ Update: Someone is now working on a Linux LiveCD to turn any idle x86 into a distcc cross-compiling machine! ]

For pure speed, you can’t beat cross-compiling directly on the big machine. But for convenience when speed isn’t absolutely critical, cross-compiling with distcc rocks!