Using the GCC regression test suite for LLVM (and other compilers)

LLVM has two main test suites:

- the regression test suite tests the compilation from source to IR; and
- the nightly test suite is a body of often large applications which are compiled and executed.

However there is no large body of tests of detailed functionality which is compiled right down to the target object code and executed. We describe the changes we have made to allow the GCC test suite to be used for nightly public regression testing of LLVM for RISC-V.

Presented at the GNU Tools Cauldron 2018 in Manchester, UK

Transcript

1. Using the GCC regression test suite for LLVM (and other compilers) Simon Cook Copyright © 2018 Embecosm. Freely available under a Creative Commons license.
2. • First spoke about this at GNU Tools Cauldron 2015 – Jeremy Bennett: Keeping other compilers honest • Aim to use relevant tests from GCC test suite to validate ports of LLVM (and other compilers) – Mainly gcc.c-torture and gcc.dg suites Background
3. • Given a list of known incompatible tests, either: – Move incompatible tests from the source tree; and/or – Provide list to DejaGNU to mark tests as XFAIL • List generated from running complete test run against golden reference of target compiler • Pros: Quick path to 0 unexpected results, fast test execution time • Cons: Relies on correctness of golden reference In 2015 - Blacklisting
4. Annotate all tests with tests to check for GCC features other compilers may not support Example __attribute__((leaf)) proc check_effective_target_leaf {} { return [check_no_compiler_messages leaf assembly { void __attribute__((leaf)) foo (void) {} }] } Annotate test with /* { dg-require-effective-target leaf } */ Feature Tests
5. Python script to manage execution of tests without build directory Similar to what make check-gcc does, but also: 1. Load balances each thread based on test count 2. Sets up any gdbservers / external tools needed for each test thread Currently configured for a given target, but plan is to load that configuration in at test time Test Runner
6. Embecosm Build Server https://buildbot.embecosm.com/view/LLVM-GCC/
7. Test suite available at https://github.com/embecosm/gcc-for-llvm-testing Latest test results: Ongoing work Plan is to finish annotating tests, then fix bugs test suite exposes Eventually, plan to merge changes upstream Current Status RISC-V # of expected passes 27750 # of unexpected failures 4083 # of unexpected successes 2 # of expected failures 14 # of unresolved testcases 1525 # of unsupported tests 3272 x86_64 # of expected passes 30882 # of unexpected failures 3016 # of unexpected successes 2 # of expected failures 15 # of unresolved testcases 183 # of unsupported tests 2956
8. Thank you Copyright © 2018 Embecosm. Freely available under a Creative Commons license.