Linux Kernel Cross Compilation
There are several reasons to cross-compile the Linux kernel:
- Compiling on a native CPU and hardware requires a wide range of devices, which is not so practical. Furthermore, the actual hardware is often not suitable for the workload of kernel compilation due to a slow CPU, small memory, etc.
- Hardware emulation (e.g., with qemu ) can be a viable substitution if the slowness can be tolerated. However, how to setup the compilation environment is another challenge, as it requires at least a preferably Linux-flavored OS, the bash and make ecosystem, and most importantly, the toolchain.
- Cross-compiling on a powerful host enables quick detection of kernel compile errors and config errors, as shown in the stable queue builds project. Coupled with emulation, testing on non-native architectures becomes easier as well.
For my personal use, I would like to see the kernel build process on each architecture, capture the compilation flags of each files, collect their linking information, and finally see if I can mine some insights out of it. In this case, given the complexity of parsing the Kconfig and Makefile , probably the best way is to actually build the kernel and dump the verbose version of the build log.
Fortunately, with a modern Linux release (like Ubuntu or Fedora), all we need to cross-compile the kernel is a toolchain that can produce binaries for another CPU architecture.
Toolchain
On cross-compiling the kernel, we only need two things: binutils , and gcc , and there are generally two ways to obtain them:
Installing pre-compiled packages can be a hassle-free way if you don’t want to get your hands dirty. In fact, for the following architectures: arm, aarch64, hppa, hppa64, m68k, mips, mips64, powerpc, powerpc64, s390x, sh4, sparc64, you can directly obtain them via apt-get from the official repositories. For example,
apt-get install binutils-aarch64-linux-gnu gcc-aarch64-linux-gnu
Toolchains for other architectures such as blackfin, c6x, and tile can be obtained from the Fedora repo and converted to .deb packages as shown in the tutorial in 2013.
However, in case you want to compile the toolchain from source, for whatever reasons you might have, such as using the latest version or a customized version of gcc , you might follow the following steps.
As a head up, I am running a standard Ubuntu 16.04.3 LTS release with kernel version 4.4.0. The host gcc and binutils (i.e., the gcc and binutils used to build the toolchain) are the default ones with the Ubuntu release, which is in version 5.4.0 and 2.27 respectively. However, I don’t see major obstacles in applying it to other combinations of OS, gcc , and binutils versions as long as they are recent enough to build the toolchain.
Before diving into the building process, let’s setup some environment variables. Part of them are for convenience reasons, while others are necessary in the build process.
export TARGET=aarch64-unknown-linux-gnu # replace with your intended target export PREFIX="$HOME/opt/cross/aarch64" # replace with your intended path export PATH="$PREFIX/bin:$PATH"
The TARGET variable should be a target triplet which is used by the autoconf system. Their valid values can be found in the config.guess script.
The last step is necessary. By adding the installation prefix to the the PATH of the current shell session, we ensure the gcc is able to detect our new binutils once we have built them.
binutils
The source code for binutils can be obtained from the GNU servers:
# for stable releases BINUTILS_VERSION=2.29.1 wget ftp://ftp.gnu.org/gnu/binutils/binutils-$BINUTILS_VERSION.tar.xz # for development branch git clone git://sourceware.org/git/binutils-gdb.git
After that, modify the source code as you wish and build it with the following steps.
cd $BINUTILS_BUILD $BINUTILS_SOURCE/configure \ --target=$TARGET \ --prefix=$PREFIX \ --with-sysroot \ --disable-nls \ --disable-werror make make install
Since we have multiple targets to build, it is better to have a separate build directory for each target.
—disable-nls tells binutils not to include native language support. This is optional, but reduces dependencies and compile time.
—with-sysroot tells binutils to enable sysroot support in the cross-compiler by pointing it to a default empty directory. By default the linker refuses to use sysroots for no good technical reason, while gcc is able to handle both cases at runtime.
—disable-werror behaves exactly as the name suggests, do not add -Werror in the flag.
After installation, the binaries aarch64-unknown-linux-gnu- should exist in $PATH .
These instructions apply to binutils version 2.29.1, which is the latest release at the time of writing.
gcc
Similar to binutils . the source code for gcc can be obtained from the GNU servers too:
# for stable releases GCC_VERSION=7.2.0 wget ftp://ftp.gnu.org/gnu/gcc/gcc-$GCC_VERSION/gcc-$GCC_VERSION.tar.xz # for development branch git clone git://gcc.gnu.org/git/gcc.git
After that, modify the source code as you wish and build it with the following steps.
cd $GCC_SOURCE ./contrib/download_prerequisites cd $GCC_BUILD $GCC_SOURCE/configure \ --target=$TARGET \ --prefix=$PREFIX \ --enable-languages=c,c++ \ --without-headers \ --disable-nls \ --disable-shared \ --disable-decimal-float \ --disable-threads \ --disable-libmudflap \ --disable-libssp \ --disable-libgomp \ --disable-libquadmath \ --disable-libatomic \ --disable-libmpx \ --disable-libcc1 make all-gcc make install-gcc
Downloading the pre-requisites (gmp, mpfr, mpc, isl) areis necessary as gcc needs these packages for compilation. This is handled seamlessly with the download_prerequisites script.
—enable-languages=c,c++ tells gcc not to build frontends for other languages like Fortran, Java, etc.
—without-headers tells gcc not to rely on any C library being present for the target.
—disable- tells gcc not to build those packages as they will not be needed in kernel compilation.
More importantly, we should not simply make all as that would build way too much (for example, the libgcc, libc, libstdc++, etc), which are not needed at all. All we need is the compiler itself which can be built by make all-gcc .
Another important note is that in order for gcc to lookup the correct set of binutils , both $TARGET and $PREFIX must be exactly the same when configuring binutils and gcc . For example, aarch64-unknown-linux-gnu-gcc will lookup aarch64-unknown-linux-gnu-as in the same directory: merely putting aarch64-unknown-linux-gnu-as in $PATH is not enough.
The instructions apply to gcc version 7.2.0, which is the latest release at the time of writing.
After the whole process, the following files should present in the $PREFIX/bin directory, and also in the $PATH .
aarch64-unknown-linux-gnu-addr2line aarch64-unknown-linux-gnu-ar aarch64-unknown-linux-gnu-as aarch64-unknown-linux-gnu-c++ aarch64-unknown-linux-gnu-c++filt aarch64-unknown-linux-gnu-cpp aarch64-unknown-linux-gnu-elfedit aarch64-unknown-linux-gnu-g++ aarch64-unknown-linux-gnu-gcc aarch64-unknown-linux-gnu-gcc-7.2.0 aarch64-unknown-linux-gnu-gcc-ar aarch64-unknown-linux-gnu-gcc-nm aarch64-unknown-linux-gnu-gcc-ranlib aarch64-unknown-linux-gnu-gcov aarch64-unknown-linux-gnu-gcov-dump aarch64-unknown-linux-gnu-gcov-tool aarch64-unknown-linux-gnu-gprof aarch64-unknown-linux-gnu-ld aarch64-unknown-linux-gnu-ld.bfd aarch64-unknown-linux-gnu-nm aarch64-unknown-linux-gnu-objcopy aarch64-unknown-linux-gnu-objdump aarch64-unknown-linux-gnu-ranlib aarch64-unknown-linux-gnu-readelf aarch64-unknown-linux-gnu-size aarch64-unknown-linux-gnu-strings aarch64-unknown-linux-gnu-strip
Kernel Build
With the toolchain ready, cross-compiling the kernel involves two extra steps:
- Find the architecture name ( $KERNEL_ARCH ) in kernel source tree
- they are typically Located in arch/* in the kernel source tree.
- Find the configuration ( $KERNEL_CONF ) for each sub-arch (e.g., 32-bit vs 64-bit, big endian vs little endian, etc)
- make ARCH=$KERNEL_ARCH help will show some hints.
- if there is no specific machine config, the defconfig should work.
- if there are available machine configs, choosing from an existing config seems to be more hassle free compared with doing a manual menuconfig .
Once we find the values for $KERNEL_ARCH and $KERNEL_CONF , cross-compiling the kernel is as easy as the following:
cd $KERNEL_SOURCE make ARCH=$KERNEL_ARCH O=$KERNEL_BUILD $KERNEL_CONF cd $KERNEL_BUILD make ARCH=$KERNEL_ARCH CROSS_COMPILE=$TARGET- V=1 vmlinux modules
For example, in the case of building the aarch64 kernel, the command should look like the following:
cd $KERNEL_SOURCE make ARCH=arm64 O=$KERNEL_BUILD defconfig cd $KERNEL_BUILD make ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- V=1 vmlinux modules
Of course we can always speed up the build process with the make -j flags.
The instructions apply to Linux kernel version 4.13.5, which is the latest release at the time of writing.
Results
In total, I have currently cross-compiled the kernel for 7 architectures and 12 sub-archs, with the procedure described above. The result is summarized in the following table:
| Architecture | Name | $TARGET | $KERNEL_ARCH | $KERNEL_CONF |
|---|---|---|---|---|
| x86 (32-bit) | i386 | i386-pc-linux-gnu | x86 | i386_defconfig |
| x86 (64-bit) | x86_64 | x86_64-pc-linux-gnu | x86 | x86_64_defconfig |
| arm (32-bit) | arm | armv7-unknown-linux-gnueabi | arm | multi_v7_defconfig |
| arm (64-bit) | aarch64 | aarch64-unknown-linux-gnu | arm64 | defconfig |
| powerpc (32-bit) | ppc | powerpcle-unknown-linux-gnu | powerpc | pmac32_defconfig |
| powerpc (64-bit) | ppc64 | powerpc64le-unknown-linux-gnu | powerpc | ppc64le_defconfig |
| sparc (32-bit) | sparc | sparc-unknown-linux-gnu | sparc | sparc32_defconfig |
| sparc (64-bit) | sparc64 | sparc64-unknown-linux-gnu | sparc | sparc64_defconfig |
| mips (32-bit) | mips | mips-unknown-linux-gnu | mips | 32r6_defconfig |
| mips (64-bit) | mips64 | mips64-unknown-linux-gnu | mips | 64r6_defconfig |
| s390x (64-bit) | s390x | s390x-ibm-linux-gnu | s390 | default_defconfig |
| ia64 (64-bit) | ia64 | ia64-unknown-linux-gnu | ia64 | generic_defconfig |
I have also tried building an allyesconfig kernel and succeeded in 7 architectures, as shown in the following table. Unfortunately, ia64 failed miserably with error message Error: Operand 2 of ‘adds’ should be a 14-bit integer (-8192-8191) , and the error seems to have been there for more than 7 months.
| Architecture | Name | $TARGET | $KERNEL_ARCH | $KERNEL_CONF |
|---|---|---|---|---|
| x86 (64-bit) | x86_64 | x86_64-pc-linux-gnu | x86 | allyesconfig |
| arm (32-bit) | arm | armv7-unknown-linux-gnueabi | arm | allyesconfig |
| arm (64-bit) | aarch64 | aarch64-unknown-linux-gnu | arm64 | allyesconfig |
| powerpc (64-bit) | ppc64 | powerpc64-unknown-linux-gnu | powerpc | allyesconfig |
| sparc (64-bit) | sparc64 | sparc64-unknown-linux-gnu | sparc | allyesconfig |
| mips (64-bit) | mips64 | mips64-unknown-linux-gnu | mips | allyesconfig |
| s390x (64-bit) | s390x | s390x-ibm-linux-gnu | s390 | allyesconfig |
In case you want to try out, you can directly feed the corresponding values for $TARGET , $KERNEL_ARCH , and $KERNEL_CONF into the scripts above and test.
Future works
I did not try with the less common (or inactive) architectures such as blackfin, sh, or tile although I believe that they can be cross-compiled in the similar way as well.
Certain architectures are not available in the official gcc source tree and one example is the gcc for openrisc. In these cases, we have to compile the toolchain from the correspondingly ported versions such as GCC port for OpenRISC 1000.
How do I cross-compile the kernel on a Ubuntu host?
I would like to understand more about how the kernel works. Part of this is to compile it myself. How do I cross-compile the Kernel on a Ubuntu host?
3 Answers 3
Preparation
First, we need to install the required prerequisites. I assume you have sudo access.
sudo apt-get install git ncurses-dev make gcc-arm-linux-gnueabi - git is the version control system used by the Linux kernel team.
- ncurses is a library for build console menus. It is necessary for menuconfig .
- make runs the compilation for us.
- gcc-arm-linux-gnueabi is the cross-compiler.
Next, we need to retrieve the source, run:
git clone https://github.com/raspberrypi/linux raspberrypi-linux cd raspberrypi-linux This will clone the source code to a directory called raspberrypi-linux and change to it.
Compilation
We first need to move the config file by running
cp arch/arm/configs/bcmrpi_cutdown_defconfig .config Then configure the kernel build
make ARCH=arm CROSS_COMPILE=/usr/bin/arm-linux-gnueabi- oldconfig Optional: Customise the build using menuconfig
make ARCH=arm CROSS_COMPILE=/usr/bin/arm-linux-gnueabi- menuconfig make ARCH=arm CROSS_COMPILE=/usr/bin/arm-linux-gnueabi- -k