openQA - Avoiding Disasters of Biblical Proportions

openQA - Avoiding Disasters of Biblical Proportions

What do we mean, biblical? 40 years of darkness, earthquakes, volcanos, human sacrifice, mass hysteria, dogs and cats living together... and on that note, did you know that openQA is used by both openSUSE and Fedora? What is so awesome about this tool that traditional distribution enemies are now working together?

openQA is an automated testing tool, capable of full system, console, and graphical application testing. This session will give an introduction to openQA's capabilities, share how it is used by openSUSE for the testing of Tumbleweed and Leap, and suggest ideas to attendees as how it could be used for testing their software, operating systems, or virtual machine images.


Richard Brown

June 24, 2016


  1. openQA Avoiding Disasters of Biblical Proportions Richard Brown – openSUSE

  2. 3 CC-BY Sophie47 @

  3. 4

  4. 5 ©

  5. 6 CC-BY Lauri Kosonen @

  6. What the ****?

  7. 8 “Testing is HARD” - Everyone who has built any

    software, ever.
  8. 9 Upstreams Upstream projects are moving fast • Kernel 4.x

    – New version every 3 months • GNOME 3.x – New version every 6 months • KDE Plasma 5.x – New version every 3 months
  9. 10 Users Users expectations are trending towards receiving software faster

    and faster • Rolling Releases (openSUSE Tumbleweed, Arch) • OBS, Copr, PPAs, AUR
  10. 11 Diversification Distribution Projects are offering more stuff in different

    ways • Editions (Fedora Workstation, Server, Cloud) • Multiple Distributions (openSUSE Leap, Tumbleweed) • Modularization, Containers, Docker, JeOS
  11. 12

  12. 13 But.. DevOps CI Containers Gerrit Jenkins

  13. 14 But.. DevOps CI Containers Gerrit Jenkins Docker

  14. 15 But.. DevOps CI Containers Gerrit Jenkins Docker Docker

  15. 16 But.. DevOps CI Containers Gerrit Jenkins Docker Docker Docker

  16. 17 But.. DevOps CI Containers Gerrit Jenkins Docker Docker Docker

    Docker Docker Docker Docker Docker
  17. 18 NEWS FLASH: We do not write code • We

    build Distributions • Our users do not care if our code works, only if it works correctly as part of the Distribution • Lots of testing tools for Code, none for actual Operating Systems and Apps as a cohesive product
  18. openQA to the rescue

  19. 20 openQA • Open Source distribution testing framework • Started

    in 2009 • Tests Operating Systems and their applications • GUI & Console Testing • Used by openSUSE® Leap, Tumbleweed, SUSE® Linux Enterprise & Fedora®
  20. 21 Tests like a User • Does NOT touch the

    software directly • Controls keyboard & mouse and uses them like a user • Uses OpenCV to ‘read’ the actual screen output and compare to predefined needles • Also reads plain text from serial
  21. 22

  22. 23 Needles only match areas of interest

  23. 24 Not just graphical testing openQA supports executing console test

    scripts on a system under test • Write scripts in the openQA DSL. • Add scripts (bash, perl, python, …) to the openQA test suite. openQA test API can deploy & execute. • Download existing script/packages then execute in the openQA DSL.
  24. 25 Supported Console Outputs Multiple methods of comparing text results

    for openQA systems under test • Simple string comparison. • JUnit Parsing. • Custom results analysis in the openQA DSL.
  25. 26 Console Example

  26. 27 openQA does NOT only test graphically

  27. 28 Reports like a Pro • Records screenshots & logs

    for analysis • Encodes video of each test run • Provides ISO/HDD images for reproduction • Dashboard for easy review and reporting
  28. 29 Product Dashboard

  29. 30 Build Dashboard

  30. 31 Test Results

  31. 32 Visual Dif

  32. 33 Visual Dif

  33. 34 Visual Dif

  34. 35 Logs & Assets

  35. 36 openQA Architecture

  36. 37 Feature Highlights • Multi Arch Support (Intel, ppc64le, s390x,

    aarch64) • FedMsg Support • Multi Machine Testing (incl. openvswitch) • Real Hardware Testing (IPMI & IP KVM) • Disk Image Creation • Testing without OS Installation (Testing from Image) • Multi Hypervisor Support (KVM, Xen, VMware, KVM for zSystems, Hyper-V)
  37. 38 Tag ALL THE THINGS!!! • New Review Workflow ‒

    Tag issues with comments or bug ID’s ‒ Tags appear next to scenario traffic lights on dashboard ‒ Repeated issues automatically inherit previous Tags • Build Tagging ‒ Mark a build as important to ensure result & asset retention ‒ Tagged builds can be immune from ‘obsolescence’
  38. 39 Build Dashboard

  39. Fedora and openQA

  40. 41 Losing my religion

  41. 42 The Glorious Present • ~40% validation tests automated •

    openQA results reported to wiki • Tests triggered automatically
  42. 43 Nightly testing • Rawhide and Branched ‘nightly’ composes tested

    • Email result notification • Nightly image finder • ‘Gating’ plans in discussion with release engineering • Easy bug reporting (installer crashes)
  43. 44 What is it good for?

  44. 45 New Directions • Testing ‘updates images’ • Kickstart-tests converter

    • Testing installer updates • Testing as a service
  45. 46 openQA in Fedora • Packaged in official repositories (F23+)

    • Two deployments (staging and production) • Both run on Fedora 23 in Fedora data center • Deployment completely Ansible-driven • Can be rebuilt from scratch at will • Ansible plays usable outside Fedora infra • Fedmsg-driven scheduler
  46. 47 Together in perfect harmony • Asset downloading and extraction

    • Fedmsg emitter • Dockerfiles • Packaging / automated deployment improvements • Searching improvements • Bug fixes • Bugs
  47. openSUSE and openQA

  48. 49 openQA & SUSE Linux Enterprise Package Build Product Milestone

    Submission Validation Alpha/Beta/RC “Staging” “Post-Validation” openQA
  49. 50 2 Codebases – 1 Distribution SUSE® Linux Enterprise

  50. 51 openSUSE Leap • openSUSE Leap is a new distribution

    combining packages from both SUSE Linux Enterprise and openSUSE • Extensive integration challenge, combining two separate code bases and ensuring key functionality is preserved • Dozens of different installation/upgrade scenarios tested for each development build

  52. 53 openSUSE Tumbleweed • Tumbleweed is openSUSE's rolling release •

    openQA is an integral part of Tumbleweed development • In depth testing is then carried out on the whole distribution before release • openSUSE Tumbleweed snapshots released after successful testing • New Tumbleweed release every 2-3 days, often faster.
  53. 54 A 'quiet' Tumbleweek • 3 Snapshots • 146 Package

    Updates • 15 New Packages on the DVD • 38 Packages Removed from the DVD • 1 new Kernel • All tested with over 118 different installations/upgrades
  54. 55 A 'quiet' Tumbleweek • 3 Snapshots • 146 Package

    Updates • 15 New Packages on the DVD • 38 Packages Removed from the DVD • 1 new Kernel • All tested with over 118 different installations/upgrades QUIET?!
  55. 56 Tumbleweek 31/03 – 7/04 2016 • 5 Snapshots •

    370 package updates • 2 new Kernels • GNOME 3.20 a week after it’s upstream release • All tested with over 118 different installations and upgrades
  56. 57 All tested by openQA Shared Core • >8000 Packages

    • Community Developed • Rolling Updates • Rolling Base System • Over 6000 Packages • Community Developed SUSE® Linux Enterprise • Enterprise Packages • SUSE Developed • Stable Base System • Regular Updates Tumbleweed
  57. Futures

  58. 59 On the drawing board • Improved IPMI backend (SuperMicro,

    Dell iDRAC, etc) • Improved “Interactive Mode” ‒ Help developers / QA reproduce bugs by giving them easy direct access to openQA systems under test • OpenStack Backend ‒ Use SUSE Cloud, and other OpenStack implimentations as a platform for openQA operating system testing • Kubernetes Backend? • Docker Backend? DockerDockerDockerDockerDockerDockerDockerDockerDocker
  59. 60 Whatever you want..or build! • Main Website ‒

    • Documentation ‒ • Bug Reports & Feature Requests ‒
  60. Questions?

  61. Thank you. Want to learn more?

  62. 63 Have a Lot of Fun, and Join Us At:
  63. General Disclaimer This document is not to be construed as

    a promise by any participating organisation to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. openSUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for openSUSE products remains at the sole discretion of openSUSE. Further, openSUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All openSUSE marks referenced in this presentation are trademarks or registered trademarks of SUSE LLC, in the United States and other countries. All third-party trademarks are the property of their respective owners. License This slide deck is licensed under the Creative Commons Attribution-ShareAlike 4.0 International license. It can be shared and adapted for any purpose (even commercially) as long as Attribution is given and any derivative work is distributed under the same license. Details can be found at Credits Template Richard Brown Design & Inspiration openSUSE Design Team guidelines/