Building Linux Image for Working with eBPF in QEMU

Overview
Preparing Linux Host
- Notes for CentOS 7
Preparing Tools and Libraries
Preparing Building Environment
Building Images
- Building rootfs.cpio
- Building vmlinux
Workarounds for Well Known Pitfalls
- Errors While Building Kernel’s eBPF Files
- Workaround for complex float error
Running Linux Image Using QEMU
Building and Running eBPF Programs
Using NFS for Building eBPF Programs Right on the Target

Overview 

This is a comprehensive guide about creating an environment for building, running and debugging eBPF programs for ARC processors using GNU toolchain and QEMU. Though we consider ARC HS 3x/4x on QEMU as a reference platform, the same guide is applicable for boards like HS Development Kit.

This guide consists of these steps:

Preparing your Linux host for building rootfs (Buildroot), Linux kernel, third-party tools and libraries.
Building and installing third-party tools and libraries: elfutils, pahole and bpftool. We are going to build them manually to ensure that the latest versions are used.
Preparing the building environment: cloning all necessary repositories, configuring SSH keys, etc.
Building rootfs (Buildroot) image and the Linux kernel.
Building and running eBPF programs.

Preparing Linux Host 

We assume that toolchain directory for ARC HS 3x/4x is placed in /tools/arc-linux-gnu (the directory which contains bin). Ensure that /tools/arc-linux-gnu/bin is in PATH environment variable. We are going to use /tools directory for installing tools and libraries. You can use any other path, just don’t forget to consider it while reading this guide.

The latest release may be downloaded here (“Linux/glibc ARC HS” variant):

https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases

Standard development tools must be installed on your host: make, cmake, git, rsync, gcc, binutils, clang (for building eBPF programs).

Notes for CentOS 7 

It’s necessary to install the latest available development tools for CentOS 7 to make it possible to build everything without problems. Use centos-release-scl repository to install the latest tools and Git. Then, have them enabled.

sudo yum install centos-release-scl
sudo yum install devtoolset-9 rh-git227
scl enable devtoolset-9 rh-git227 bash

Preparing Tools and Libraries 

We are going to build and install some tools an libraries manually:

1. pahole host tool is used during the generation of BTF information for the Linux image. We have to use version ≤1.23 because later versions generate BTF information for 64-bit enumerations. However, the Linux kernel of version ≤6.0 contains tools which don’t support such BTF records and building fails on the last stage. We need to ensure that a proper pahole is used.

2. elfutils host libraries of version ≥0.189 must be presented in LD_LIBRARY_PATH because pahole relies on them for working with binaries. Support of ARCv2 was added in elfutils 0.189, thus we need to ensure that pahole is linked with recent-enough elfutils libraries.

3. bpftool of version ≥7 must be used for building eBPF program which use features like bpf_loop, calls to another functions, etc. Older versions don’t support new type of relocations for such features. If you are experiencing problems with host’s bpftool (e.g., Ubuntu 22.04 is shipped with an outdated bpftool which may fail while building the kernel) then it would be better to build and install it manually.

Building and Installing `elfutils`

# Install dependencies for CentOS 7
sudo yum install libmicrohttpd libmicrohttpd-devel libsq3 libsq3-devel \
                 libarchive libarchive-devel gettext-devel zstd libcurl-devel

# Install dependencies for the latest Fedora
sudo dnf install libmicrohttpd libmicrohttpd-devel libsq3 libsq3-devel \
                 libarchive libarchive-devel gettext-devel

# Install dependencies for Ubuntu 18.04
sudo apt install libmicrohttpd-dev libsqlite3-dev libarchive-dev

# Clone, configure and build elfutils (use your own prefix instead of /tools/elfutils)
git clone -b elfutils-0.189 https://sourceware.org/git/elfutils.git
cd elfutils
autoreconf -fi
mkdir build
cd build
../configure --prefix=/tools/elfutils --enable-maintainer-mode
make
make install

# Configure your environment
export PATH=/tools/elfutils/bin:$PATH
export LD_LIBRARY_PATH=/tools/elfutils/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}

Building and Installing `pahole`

# Clone, configure and build pahole (use your own prefix instead of /tools/pahole)
git clone -b v1.23 https://git.kernel.org/pub/scm/devel/pahole/pahole.git
mkdir pahole/build
cd pahole/build
cmake -G "Unix Makefiles"                              \
      -D__LIB=lib                                      \
      -DDWARF_INCLUDE_DIR=/tools/elfutils/include      \
      -DLIBDW_INCLUDE_DIR=/tools/elfutils/include      \
      -DDWARF_LIBRARY=/tools/elfutils/lib/libdw.so.1   \
      -DELF_LIBRARY=/tools/elfutils/lib/libelf.so.1    \
      -DCMAKE_INSTALL_PREFIX=/tools/pahole             \
      ..
make install

# Configure your environment
export PATH=/tools/pahole/bin:$PATH
export LD_LIBRARY_PATH=/tools/pahole/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}

Building and Installing `bpftool`

# Clone and build bpftool (use your own prefix instead of /tools/bpftool)
git clone --recurse-submodules https://github.com/libbpf/bpftool.git
cd bpftool/src
make prefix=/tools/bpftool EXTRA_CFLAGS="-I/tools/elfutils/include" \
                           EXTRA_LDFLAGS="-L/tools/elfutils/lib"    \
                           install-bin

# Configure your environment
export PATH=/tools/bpftool/sbin/:$PATH

Preparing Building Environment 

Cloning ARC eBPF Testbench 

Clone ARC eBPF testbench. This repository contains configuration files for Buildroot and Linux kernel which simplify setup of the environment for working with eBPF. We are going to use it as a working directory.

git clone --recurse-submodules https://github.com/foss-for-synopsys-dwc-arc-processors/arc-bpf-testbench
cd arc-bpf-testbench

Preparing Buildroot 

Clone Buildroot, create a build directory and copy all necessary configuration files and an overlay to the build directory from arc-bpf-testbench/extras:

git clone https://git.buildroot.net/buildroot
mkdir buildroot/build
cp -r extras/buildroot/* buildroot/build

List of copied files and directories:

busybox.fragment - A configuration file for BusyBox.
device_table.txt - A configuration file fo setting proper permissions for files in the overlay.
qemu_hs4x_ebpf_defconfig - A configuration file for Buildroot.
overlay - All necessary additional files for target’s file system (configuration files, testing SSH keys, etc.).

It’s assumed here that the root directory for the toolchain is /tools/arc-linux-gnu. Thus you need to change BR2_TOOLCHAIN_EXTERNAL_PATH in qemu_hs4x_ebpf_defconfig configuration file for Buildroot to the corresponding path.

Preparing Linux Sources 

Clone repository of the Linux kernel with the latest patches for support of eBPF with JIT and copy a corresponding configuration file:

git clone -b bpf-early-access https://github.com/foss-for-synopsys-dwc-arc-processors/linux
mkdir linux/build
cp extras/linux/qemu_hs4x_ebpf_defconfig linux/arch/arc/configs

Installing SSH Keys for User’s Authentication 

We are going to use SSH for interacting with the ARC Linux system. It would be helpful to have keys for public key authorization without using a password.

You can copy the pregenerated keys from extras/host/.ssh/keys to the corresponding host’s directory ~/.ssh/keys. Public key for this pair of keys is already installed in the buildroot/build/overlay/root/.ssh directory. Don’t forget to apply proper rights for those keys in .ssh for your host (600).

Configure your SSH hosts in ~/.ssh/config (you also can find this file in extras/host/.ssh/config):

Host arc
    HostName            127.0.0.1
    Port                2022
    User                root
    IdentityFile        ~/.ssh/keys/arc

Host arc-tap
    HostName            10.42.0.100
    Port                22
    User                root
    IdentityFile        ~/.ssh/keys/arc

Also, you can generate your own keys (use your own home path):

$ mkdir -p ~/.ssh/keys
$ ssh-keygen -t rsa -C "arc@ebpf"
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa): /home/user/.ssh/keys/arc
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user/.ssh/keys/arc
Your public key has been saved in /home/user/.ssh/keys/arc.pub

Add your public key to the overlay directory:

mkdir -p buildroot/build/overlay/root/.ssh
cp -f ~/.ssh/keys/arc.pub buildroot/build/overlay/root/.ssh/authorized_keys

Install SSH Keys for Host’s Authentication (Host Keys)

By default, Linux with SSH daemon installed generates random host keys if they don’t exist. For testing and debugging purposes using QEMU it may lead to these difficulties:

Generating a set of host keys in QEMU may take a lot of time.
Each time you run QEMU with vmlinux image new keys a generated. Thus, you have to clear cached host key for the QEMU instance to avoid complaining about the changed target’s host key.

Overlay already contains pregenerated host keys. However, you can generate your own keys:

ssh-keygen -A -f buildroot/build/overlay

Building Images 

Building `rootfs.cpio`

Warning

Buildroot requires Git of version 2+. Some old systems (e.g., CentOS 7) have an outdated Git which is not supported by Buildroot’s build system. If you face this problem then you have to find a way to install newer version of Git (e.g., using third-party repositories).

Note

SSHFS package requires docutils module for Python. Install it using your package manager or using pip (pip install docutils) or delete the BR2_PACKAGE_SSHFS=y line if you aren’t going to use SSHFS.

Note

Buildroot may complain about invalid headers’ version for the toolchain:

Incorrect selection of kernel headers: expected 5.16.x, got 5.18.x

E.g., 2022.09 release is shipped with headers for Linux kernel 5.16.x. If it’s not your case then manually change headers’ versions for the toolchain using make menuconfig.

cd buildroot/build
make -C .. O=$(pwd) defconfig BR2_DEFCONFIG=build/qemu_hs4x_ebpf_defconfig
make -j $(nproc)

Building `vmlinux`

# Set necessary environment variables and build the kernel.
cd ../../linux/build
export ARCH=arc
export CROSS_COMPILE=arc-linux-gnu-
export C_INCLUDE_PATH="/tools/elfutils/include"
export LIBRARY_PATH="/tools/elfutils/lib"
make -C .. O=$(pwd) qemu_hs4x_ebpf_defconfig
make -j $(nproc)

Workarounds for Well Known Pitfalls 

Errors While Building Kernel’s eBPF Files 

Change kernel/bpf/Makefile to prevent some build errors:

diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index ae90af5b0425..4699a022079a 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -4,7 +4,7 @@ ifneq ($(CONFIG_BPF_JIT_ALWAYS_ON),y)
# ___bpf_prog_run() needs GCSE disabled on x86; see 3193c0836f203 for details
cflags-nogcse-$(CONFIG_X86)$(CONFIG_CC_IS_GCC) := -fno-gcse
endif
-CFLAGS_core.o += $(call cc-disable-warning, override-init) $(cflags-nogcse-yy) -Og -g3
+CFLAGS_core.o += $(call cc-disable-warning, override-init) $(cflags-nogcse-yy) -Og -g3 -finline-functions-called-once

Workaround for `complex float` error 

Toolchains for ARC generate complex float DIE entries in libgcc. At the moment such entries are not supported by pahole. So, it’s necessary to disable generating BTF for floats. It’s already done in bpf-early-access branch but if you want to build the Linux kernel from another branch or repository with BTF information you can apply this patch (https://github.com/foss-for-synopsys-dwc-arc-processors/linux/commit/b17d1955b67493afe37430694c8982411336fc4c):

diff --git a/scripts/pahole-flags.sh b/scripts/pahole-flags.sh
index 0d99ef17e4a5..23af14c6ef94 100755
--- a/scripts/pahole-flags.sh
+++ b/scripts/pahole-flags.sh
@@ -14,7 +14,7 @@ if [ "${pahole_ver}" -ge "118" ] && [ "${pahole_ver}" -le "121" ]; then
        extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_vars"
fi
if [ "${pahole_ver}" -ge "121" ]; then
-       extra_paholeopt="${extra_paholeopt} --btf_gen_floats"
+       extra_paholeopt="${extra_paholeopt}"
fi
if [ "${pahole_ver}" -ge "122" ]; then
        extra_paholeopt="${extra_paholeopt} -j"

Running Linux Image Using QEMU 

All actions mentioned below are performed from the working directory (root of arc-bpf-testbench).

Running Linux Using User Level Network Interface 

make qemu-start

Running Linux Using TUN/TAP Network Interface 

TUN/TAP network interface allows interacting of the target with your host in both directions. For example, you can mount host’s NFS directories inside of the target. Configure TUN/TAP interface on host’s side:

# Manually
sudo ip tuntap add tap1 mode tap
sudo ip addr add 10.42.0.1/24 dev tap1
sudo ip link set tap1 up

# ... or using testbench
sudo make tap

Then run vmlinux:

make USE_TAP=1 qemu-start

Configure a network interface on target’s side:

ifconfig eth0 10.42.0.100

Configuring Linux 

Mount debugfs and turn JIT on:

# On host's side
ssh arc "mount -t debugfs debugfs /sys/kernel/debug"
ssh arc "sysctl net.core.bpf_jit_enable=1"

# ... or on target's side
mount -t debugfs debugfs /sys/kernel/debug
sysctl net.core.bpf_jit_enable=1

# ... or using testbench for user level network interface
make qemu-setup

# ... or using testbench for tun/tap network interface
make USE_TAP=1 qemu-setup

Running Kernel’s Basic eBPF Tests 

Send a module for testing to the target:

# For user level network interface
rsync linux/build/lib/test_bpf.ko arc:/root

# For TUN/TAP network interface
rsync linux/build/lib/test_bpf.ko arc-tap:/root

Run the module on the target:

# Run all tests
insmod test_bpf.ko

# Run a specific
insmod test_bpf.ko test_id=42

# Run a range of tests
insmod test_bpf.ko test_range=42,142

Building and Running eBPF Programs 

Warning

Old operating systems like CentOS 7 and Ubuntu 18.04 contain old versions of clang which may not be sufficient for building modern eBPF programs. If building eBPF programs fails then try to build the latest clang with eBPF target following a corresponding guide and put it into PATH.

Testbench contains a bunch of examples of eBPF programs. You can build and load them using these commands from the root directory of the testbench:

# Build dependencies
make

# Load programs for user level network interface
make qemu-load

# or for TUN/TAP network interface
make USE_TAP=1 qemu-load

Run a program:

# Manually on target's side
./minimal

# ... or using testbench on host's side
make run-minimal

Explore README.md for ARC eBPF Testbench or run make help for information about available commands.

Using NFS for Building eBPF Programs Right on the Target 

Configuring NFS in CentOS 7 or Fedora 

Install NFS to the host:

# For CentOS 7
sudo yum install nfs-utils

# For Fedora
sudo dnf install nfs-utils

Enable services and add rules for firewall:

sudo systemctl enable --now rpcbind nfs-server
sudo firewall-cmd --add-service=nfs --permanent
sudo firewall-cmd --reload

# Optional (only if you are going to use SSHFS instead of NFS)
sudo systemctl enable sshd
sudo systemctl start sshd

Configuring NFS in Ubuntu 18.04 

Install NFS to the host and enable it:

sudo apt install nfs-kernel-server
sudo systemctl enable --now nfs-server

Configuring `/etc/exports`

Add this line to /etc/exports (you can find anonuid and anongid for your user using id -u and id -g respectively):

/nfs *(rw,all_squash,anonuid=1000,anongid=1000,no_subtree_check,insecure)

Update the table of exported NFS file systems:

sudo exportfs -rv

Mounting NFS Directory Inside the Guest 

Create a directory for mounting NFS directory on target’s side:

mkdir /nfs

If you use user level network interface for running QEMU then just run these commands inside the guest:

# Using NFS
mount -t nfs 10.0.2.2:/nfs /nfs -o nolock

# Using SSHFS
sshfs -o idmap=user,allow_other user@10.0.2.2:/nfs /nfs

If you prefer using TUN/TAP network interface, then run QEMU like make USE_TAP=1 qemu-start and configure guest’s network interface as mentioned earlier. Then run this line on target’s side:

# Using NFS
mount -t nfs 10.42.0.1:/nfs /nfs -o nolock

# Using SSHFS
sshfs -o idmap=user,allow_other user@10.42.0.1:/nfs /nfs

Preparing Tools 

Build Clang for ARC and place a directory with clang to /nfs (the full path to Clang root directory must be /nfs/clang).

Download, unpack and place a native glibc ARC HS toolchain into /nfs/arc-linux-gnu.

Copy /tools/arc-linux-gnu/sysroot to /nfs/sysroot. Also build applications using testbench (run make from the root directory of the testbench) and copy headers to the sysroot:

cp -r output/arc/deps/include/* /nfs/sysroot/usr/include/

Copy applications from the testbench:

cp -r apps /nfs

Building eBPF Application 

Use these commands inside the ARC guest (QEMU):

# Configure your PATH
export PATH="/nfs/clang/bin:$PATH"
export PATH="/nfs/arc-linux-gnu/bin:$PATH"

# Build "minimal"
cd /nfs/apps
clang -g                         \
      -O2                        \
      -target bpf                \
      -D__TARGET_ARCH_arc        \
      -I/nfs/sysroot/usr/include \
      -c minimal.bpf.c           \
      -o minimal.bpf.o

bpftool gen skeleton minimal.bpf.o > minimal.skel.h

gcc -I/nfs/sysroot/usr/include \
    -L/usr/lib minimal.c       \
    -lbpf                      \
    -lelf                      \
    -lz                        \
    -o minimal

# Prepare the Linux kernel
mount -t debugfs debugfs /sys/kernel/debug
sysctl net.core.bpf_jit_enable=1

# Run the application
./minimal