Using the GCC regression test suite for LLVM (and other compilers)

LLVM has two main test suites:

- the regression test suite tests the compilation from source to IR; and
- the nightly test suite is a body of often large applications which are compiled and executed.

However there is no large body of tests of detailed functionality which is compiled right down to the target object code and executed. We describe the changes we have made to allow the GCC test suite to be used for nightly public regression testing of LLVM for RISC-V.

Presented at the GNU Tools Cauldron 2018 in Manchester, UK

Transcript

1. Using the GCC regression test suite for
   LLVM (and other compilers)
   Simon Cook
   Copyright © 2018 Embecosm.
   Freely available under a Creative Commons license.
   

   View Slide

2. ● First spoke about this at GNU Tools Cauldron 2015
   – Jeremy Bennett: Keeping other compilers honest
   ● Aim to use relevant tests from GCC test suite to validate ports of
   LLVM (and other compilers)
   – Mainly gcc.c-torture and gcc.dg suites
   Background
   

   View Slide

3. ● Given a list of known incompatible tests, either:
   – Move incompatible tests from the source tree; and/or
   – Provide list to DejaGNU to mark tests as XFAIL
   ● List generated from running complete test run against golden
   reference of target compiler
   ● Pros: Quick path to 0 unexpected results, fast test execution time
   ● Cons: Relies on correctness of golden reference
   In 2015 - Blacklisting
   

   View Slide

4. Annotate all tests with tests to check for GCC features other
   compilers may not support
   Example __attribute__((leaf))
   proc check_effective_target_leaf {} {
   return [check_no_compiler_messages leaf assembly {
   void __attribute__((leaf)) foo (void) {}
   }]
   }
   Annotate test with /* { dg-require-effective-target leaf } */
   Feature Tests
   

   View Slide

5. Python script to manage execution of tests without build directory
   Similar to what make check-gcc does, but also:
   1. Load balances each thread based on test count
   2. Sets up any gdbservers / external tools needed for each test
   thread
   Currently configured for a given target, but plan is to load that
   configuration in at test time
   Test Runner
   

   View Slide

6. Embecosm Build Server
   https://buildbot.embecosm.com/view/LLVM-GCC/
   

   View Slide

7. Test suite available at https://github.com/embecosm/gcc-for-llvm-testing
   Latest test results:
   Ongoing work
   Plan is to finish annotating tests, then fix bugs test suite exposes
   Eventually, plan to merge changes upstream
   Current Status
   RISC-V
   # of expected passes 27750
   # of unexpected failures 4083
   # of unexpected successes 2
   # of expected failures 14
   # of unresolved testcases 1525
   # of unsupported tests 3272
   x86_64
   # of expected passes 30882
   # of unexpected failures 3016
   # of unexpected successes 2
   # of expected failures 15
   # of unresolved testcases 183
   # of unsupported tests 2956
   

   View Slide

8. Thank you
   Copyright © 2018 Embecosm.
   Freely available under a Creative Commons license.
   

   View Slide