How to run DejaGNU tests for ARC toolchain

This article describes how to run testsuites of GNU projects for ARC. This article covers only baremetal toolchain. Tests can be run on different platforms - simulators or hardware platforms.

Prerequisites

  1. Get sources of the project you want to test.

  2. Get source of the “toolchain” repository:

    $ git clone git@github.com:foss-for-synopsys-dwc-arc-processors/toolchain.git

  3. Download free nSIM from webpage: https://www.synopsys.com/cgi-bin/dwarcnsim/req1.cgi. Note that requests are manually approved, hence it make take up to 2 workdays for download to be ready.

  4. Build toolchain.

Preparing

Create a directory where artifacts and temporary files will be created:

$ mkdir tests
$ cd tests

Create a site.exp file in this directory. toolchain/dejagnu/example_site.exp can be used as an example. Note that this file is not intended to be used as-is, it must be edited for a particular case: srcdir and arc_exec_prefix variables must be set to location of toolchain sources and location of installed toolchain respectively. Variable toolchain_sysroot_dir in that file shouldn’t be set for baremetal toolchain testing.

Required environment variables are:

  • ARC_MULTILIB_OPTIONS - should be set to compiler that to be tested. Note that first -m<something> should be omitted, but only first. So for example this variables might take value: cpu=arcem -mnorm.

  • DEJAGNU - should be set to path to the site.exp file in “toolchain” repository.

  • PATH - add toolchain installation location bin directory to PATH - some of the internal checks in the GNU testsuite often ignore tool variables set in site.exp and instead expect that tools are in the PATH.

  • NSIM_HOME - should point to location where nSIM has been untarred.

  • ARC_GCC_COMPAT_SUITE - set to 0 if you are not going to run compatibility tests.

Some actions are specific to particular GNU projects:

  • Newlib requires:

    $ mkdir targ-include
$ ln -s /home/user/arc_gnu/INSTALL/arc-elf32/include/newlib.h targ-include/

  • GCC might require (for some tests):

    $ pushd home/user/arc_gnu/gcc/
$ ./contrib/gcc_update --touch
$ popd

  • libstdc++ requires:

    $ export CXXFLAGS="-O2 -g"

  • GDB requires:

    $ testsuite=$/home/user/arc_gnu/gdb/testsuite
$ mkdir $(ls -1d $testsuite/gdb.* | grep -Po '(?<=\/)[^\/]+$')

Also arc-nsim.exp board will require an environment variable ARC_NSIM_PROPS to be set and to contain path to nSIM properties file that specifies ISA options.

Toolchain repository an example run.sh file that does some of those actions: toolchain/dejagnu/example_run.sh. Note that example file is not intended to run as-is - it should be modified for particular purpose.

Baremetal boards that run tests in the development systems, like arc-openocd.exp require a valid memory.x file in the current working directory.

Running

Now, that all is set, test suite can be run with:

$ runtest --tool=<project-to-test> --target_board=<board> \
  --target=arc-default-elf32

Where <project-to-test> can be: gcc, g++, binutils, ld, gas, newlib, libstdc++ or gdb. board for free nSIM can be arc-nsim.exp or arc-sim-nsimdrv.exp. The former runs nSIM as a GDBserver, while the latter will run nSIM as a standalone simulator, which is faster and more stable, but not suitable to run GDB testsuite.

If example_run.sh is being used, then assuming that it has been configured properly, then running is as simple as invoking it.

Compatibility tests

GCC contains a set of compatibility tests named compat.exp. It allows to test compatibility of ARC GNU gcc compiler and proprietary Synopsys MetaWare ccac compiler for ARC EM and ARC HS targets. If you want to run these tests it is necessary to configure additional variables in site.exp file:

  • set is_gcc_compat_suite "1" - enable support of compatibility tests from gcc.

  • set ALT_CC_UNDER_TEST "path/to/ccac"

  • set ALT_CXX_UNDER_TEST "path/to/ccac"

  • set COMPAT_OPTIONS [list [list "options for gcc" "options for ccac"]]

  • set COMPAT_SKIPS [list {ATTRIBUTE}] - disable tests with packed structures to avoid unaligned access errors.

Then runtest program must be invoked with an additional option compat.exp:

$ runtest --tool=<project-to-test> --target_board=<board> \
  --target=arc-default-elf32 compat.exp

Also you can use example_run.sh and example_site.exp to simplify configuration and set these environment variables in example_run.sh:

  • runtestflags - set to compat.exp to run compatibility tests only.

  • ARC_GCC_COMPAT_SUITE - set to 1.

  • GCC_COMPAT_CCAC_PATH - path to Synopsys MetaWare ccac executable.

  • GCC_COMPAT_GCC_OPTIONS - options for gcc.

  • GCC_COMPAT_CCAC_OPTIONS - options for ccac.

ARC-specific board options

Following options are supported by ARC DejaGNU scripts and are usually set in board files, for example in dejagnu/baseboard/arc-sim-nsimdrv.exp.

arc,gdbserver_prog

Path to GDB server to use with arc-nsim.exp.

arc,gdbserver_args

Argument to pass to GDB server used in arc-nsim.exp.

arc,hostlink

Hostlink type to use. Can be nsim or empty/not set.

arc,is_gcc_compat_suite

Whether this is a GCC compat testsuite or not. Boolean value.

arc,openocd_prog

Path to OpenOCD application binary.

arc,openocf_cfg

OpenOCD configuration file. Passed to openocd via option -s as-is.

arc,openocd_log

Path to logfile for OpenoCD.

arc,openocd_log_level

Level of OpenOCD verbosity. Integer from 0 to 3 inclusive.