UsingDistcc

Using distcc to compile FatdogArm applications with Fatdog64

Cross-compiling is fast but problematic, not all software will cross-compile correctly. Native compiling is easier from this standpoint but unless one has multi-core ARM CPU with multi-gigabyte RAM, native compiling is slow (ok, actually even with the multi-core multi-gigabyte ARM platform, it is still slow compared to x86 platform).

Thanks to the guys from the Samba team, there is another way: distcc.

What is distcc?

According to its manpage, distcc is:

distcc distributes compilation of C code across several machines on a network. distcc should always generate the same results as a local compile, it is simple to install and use, and it is often much faster than a local compile.

So from the point of program being compiled, using distcc is (almost) identical to a "local compile", that is native compiling, which eliminates most of the cross-compile complications. But because distcc distributes (=sends) the compilation work elsewhere (=the more powerful machines), it will be a lot faster than normal native compiling.

How much faster? The compilation phase of Seamonkey 2.20 took 15 hours on Mele. Using distcc, the time taken was just a tad more over 3 hours.

This is the method used by Aboriginal Linux (native compiling in Qemu and distcc on its host). Aboriginal provides tools to automate everything - from building the cross-compiler, native-compiler, distcc and connecting all of them together, as well as ready-made system-images, cross- and native- compilers (which we used in FirstBoot and NativeCompiler) and all the glue scripts.

Using distcc on FatdogArm and Fatdog64

On Fatdog64

Install distcc package
Install cross-gcc-4.6.2-armv7 package
Start distccd as follows:
```
DISTCCD_PATH=/cross-tools/bin distccd --daemon --allow 0.0.0.0/0
```
Change 0.0.0.0/0 to your preferred ipaddress/netmask.

Do the above steps for all the Fatdog64 machines you want to use. Make sure that your Fatdog64 is reachable from your FatdogArm machine. Distcc uses port 3632 by default, make sure your firewall is configured to let inbound connections on this port.

Note: distccd cannot run as root (for good reasons - distccd is in fact a passwordless remote shell!). By default it will try to run as user "distcc", failing that it will try "nobody". You can add "--user spot" to tell to run as "spot", but in either case please make sure that user "spot" (or "distcc" or "nobody") has access to the cross-compiler using its primary group. If you don't like this restriction, simply "su spot" first and then run distcc from there; this way it will use the access provided by the auxiliary groups as well.

On FatdogArm

Install distcc package
Install distcc-helper package
Before you compile anything, just ensure that you export the following two environment variables:
```
export PATH=/usr/lib/distcc/bin:$PATH
export DISTCC_HOSTS="--localslots=1 --localslots_cpp=2 fatdog64-hostname,lzo"
```
Change fatdog64-hostname with the hostname or IP address of the Fatdog64 machine. If you have more than one machine configured for distcc then you can add as many hostnames as you wish.
Note: lzo means that the files are compressed with LZO first before sent to the remote server, this causes overheads on the native machine but this is usually more than offset by the resulting reduced network latency. Remove this if you don't want to use compression at all.
Run "make" with "-j" option (parallel make). As a rule of thumb, the number of concurrent jobs should be about twice the number of available server CPUs combined.

For more options, please read distcc and distccd manual pages, here: http://distcc.googlecode.com/svn/trunk/doc/web/man/

About the cross-compiler

If you don't want (or can't) use the provided gcc-cross-4.6.2-armv7 package from Fatdog64 repository, you can always roll on your own. Follow the steps on BuildingTheToolChain to create your own cross-compiler. The constraint is that the compiler you use must be identical (or almost identical) to the one supplied with FatdogArm or otherwise you will experience subtle failures.

The compiler triplet must be armv7l-unknown-linux-gnueabihf (that's the name of FatdogArm's native compiler), you can change this to something else by editing /usr/lib/distcc/bin/gcc and /usr/lib/distcc/bin/g++ scripts. The scripts have (commented out) examples where the remote compiler has arm-lfs-linux-gnueabi triplet (this is what you get if you follow Cross-LFS 2.0 guide without modification (it won't work as is, you need to change gcc specs to make it default to use hardfloat ABI).

Distcc mode of operation

distcc (from version 3.x onwards) has two modes of operation: standard mode and "pump" mode.

Standard mode is relatively straightforward and it is what has just been explained. In this mode the source file is pre-processed locally, compiled and assembled remotely, and then linked locally.

In "pump" mode, the pre-processing is also done remotely, thus speeding up the process even more. However it also demands more stricter cross-compiler requirements (identical system headers; identical location of system headers, etc); which is a lot more difficult to achieve in cross-compilation environments.

The setup described in this article operates in standard mode.

Further references

http://distcc.googlecode.com/svn/trunk/doc/web/index.html