QEMU for ARM Processes

I’m currently doing some embedded work on ARM systems. Having a virtual ARM environment is of course helpful. For the i586 class embedded systems that I run it’s very easy to setup a virtual environment, I just have a chroot run from systemd-nspawn with the --personality=x86 option. I run it on my laptop for my own development and on a server my client owns so that they can deal with the “hit by a bus” scenario. I also occasionally run KVM virtual machines to test the boot image of i586 embedded systems (they use GRUB etc and are just like any other 32bit Intel system).

ARM systems have a different boot setup, there is a uBoot loader that is fairly tightly coupled with the kernel. ARM systems also tend to have more unusual hardware choices. While the i586 embedded systems I support turned out to work well with standard Debian kernels (even though the reference OS for the hardware has a custom kernel) the ARM systems need a special kernel. I spent a reasonable amount of time playing with QEMU and was unable to make it boot from a uBoot ARM image. The Google searches I performed didn’t turn up anything that helped me. If anyone has good references for getting QEMU to work for an ARM system image on an AMD64 platform then please let me know in the comments. While I am currently surviving without that facility it would be a handy thing to have if it was relatively easy to do (my client isn’t going to pay me to spend a week working on this and I’m not inclined to devote that much of my hobby time to it).

QEMU for Process Emulation

I’ve given up on emulating an entire system and now I’m using a chroot environment with systemd-nspawn.

The package qemu-user-static has staticly linked programs for emulating various CPUs on a per-process basis. You can run this as “/usr/bin/qemu-arm-static ./staticly-linked-arm-program“. The Debian package qemu-user-static uses the binfmt_misc support in the kernel to automatically run /usr/bin/qemu-arm-static when an ARM binary is executed. So if you have copied the image of an ARM system to /chroot/arm you can run the following commands like the following to enter the chroot:

cp /usr/bin/qemu-arm-static /chroot/arm/usr/bin/qemu-arm-static
chroot /chroot/arm bin/bash

Then you can create a full virtual environment with “/usr/bin/systemd-nspawn -D /chroot/arm” if you have systemd-container installed.

Selecting the CPU Type

There is a huge range of ARM CPUs with different capabilities. How this compares to the range of x86 and AMD64 CPUs depends on how you are counting (the i5 system I’m using now has 76 CPU capability flags). The default CPU type for qemu-arm-static is armv7l and I need to emulate a system with a armv5tejl. Setting the environment variable QEMU_CPU=pxa250 gives me armv5tel emulation.

The ARM Architecture Wikipedia page [2] says that in armv5tejl the T stands for Thumb instructions (which I don’t think Debian uses), the E stands for DSP enhancements (which probably isn’t relevant for me as I’m only doing integer maths), the J stands for supporting special Java instructions (which I definitely don’t need) and I’m still trying to work out what L means (comments appreciated).

So it seems clear that the armv5tel emulation provided by QEMU_CPU=pxa250 will do everything I need for building and testing ARM embedded software. The issue is how to enable it. For a user shell I can just put export QEMU_CPU=pxa250 in .login or something, but I want to emulate an entire system (cron jobs, ssh logins, etc).

I’ve filed Debian bug #870329 requesting a configuration file for this [1]. If I put such a configuration file in the chroot everything would work as desired.

To get things working in the meantime I wrote the below wrapper for /usr/bin/qemu-arm-static that calls /usr/bin/qemu-arm-static.orig (the renamed version of the original program). It’s ugly (I would use a config file if I needed to support more than one type of CPU) but it works.

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>

int main(int argc, char **argv)
  if(setenv("QEMU_CPU", "pxa250", 1))
    printf("Can't set $QEMU_CPU\n");
    return 1;
  execv("/usr/bin/qemu-arm-static.orig", argv);
  printf("Can't execute \"%s\" because of qemu failure\n", argv[0]);
  return 1;

2 comments to QEMU for ARM Processes

  • Were you able to get networking to work ?
    The last time I tried (it was around 6 months ago), I had issues with the qemu network bridge. I don’t recollect the specifics but for sure was to do with qemu networking

  • Yes. I have the work lying around in the other machine. And networking is still broken. If you were able to work around it, it’d be nice if you could share that information. Thanks. IIRC, anything other than x86 failed to network. And back then when I checked, there was some regression introduced in qemu network.

    rrs@chutzpah:~$ machinectl login sid-armhf-base
    ==== AUTHENTICATING FOR org.freedesktop.machine1.login ===
    Authentication is required to log into a local container.
    Authenticating as: Ritesh Raj Sarraf,,, (rrs)
    Connected to machine sid-armhf-base. Press ^] three times within 1s to exit session.

    Debian GNU/Linux stretch/sid deb-armhf pts/0

    deb-armhf login: root
    Last login: Sun Jan 1 19:05:12 IST 2017 on console
    Linux deb-armhf 4.10.17+ #40 SMP Wed Jul 26 23:31:22 IST 2017 armv7l

    The programs included with the Debian GNU/Linux system are free software;
    the exact distribution terms for each program are described in the
    individual files in /usr/share/doc/*/copyright.

    Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
    permitted by applicable law.
    root@deb-armhf:~# ip a
    Cannot open netlink socket: Address family not supported by protocol
    root@deb-armhf:~# ifconfig
    lo: flags=73 mtu 65536 [0/217]
    inet netmask
    inet6 ::1 prefixlen 128 scopeid 0x10
    loop txqueuelen 1000 (Local Loopback)
    RX packets 0 bytes 0 (0.0 B)
    RX errors 0 dropped 0 overruns 0 frame 0
    TX packets 0 bytes 0 (0.0 B)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

    root@deb-armhf:~# cat /proc/net/dev
    Inter-| Receive | Transmit
    face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
    host0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    lo: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    root@deb-armhf:~# dhclient host^C
    root@deb-armhf:~# cat /etc/network/interfaces
    # interfaces(5) file used by ifup(8) and ifdown(8)
    # Include files from /etc/network/interfaces.d:
    source-directory /etc/network/interfaces.d
    root@deb-armhf:~# cat /etc/network/interfaces.d/^C
    root@deb-armhf:~# dhclient host0
    Cannot open netlink socket: Address family not supported by protocol
    Unsupported setsockopt level=263 optname=8
    Connection to machine sid-armhf-base terminated.
    14:25 ♒♒♒ ☺
    rrs@chutzpah:~$ machinectl list-images
    debTemplate subvolume no 762.2M Mon 2015-09-21 23:00:48 IST n/a
    debian-sid-mips subvolume no 369.3M Sun 2016-12-18 18:28:41 IST n/a
    fedoraTemplate subvolume no 1.2G Mon 2015-09-21 23:16:24 IST n/a
    fitbit subvolume no 1.0G Wed 2017-02-15 20:15:45 IST n/a
    sid-armhf-base subvolume no 638.7M Sun 2016-12-18 19:28:39 IST n/a
    trustyTemplate subvolume no 414.8M Mon 2015-09-21 23:01:33 IST n/a
    tumbleweedTemplate subvolume no 1.8G Mon 2015-09-21 23:12:56 IST n/a

    7 images listed.
    14:26 ♒♒♒ ☺