Pushing your QA upstream

Slides for the presentation given in the OpenStack Barcelona Summit with the following abstract:

Quality Assurance (QA) is an extremely important part of the software development process nowadays. OpenStack is not different and has a growing set of tools and infrastructure to improve its quality: integration tests (tempest and tempest plugins), performance tests (rally) and a great CI (zuul). In this talk, you will learn why you should push your QA tests upstream and how to do it, including: adding a gate job, adding tests to it, creating a custom DevStack deployment and circumventing possible limitations.

- Presentation page: https://www.openstack.org/videos/video/pushing-your-qa-upstream
- Blog post related to the presentation: http://rodrigods.com/pushing-your-qa-upstream/

Transcript

1. Pushing your QA upstream Rodrigo Duarte Sousa Senior Quality Engineer October 25th, 2016
2. WHO AM I?
3. INSERT DESIGNATOR, IF NEEDED 3 Who am I? • OpenStack user since 2013 • Work with OpenStack (mostly Keystone) since 2014 ◦ Core reviewer of Oslo.policy ◦ Core reviewer of Keystone • Currently, hunt OpenStack bugs at Red Hat rodrigodsousa (Twitter), rodrigods (GitHub, IRC)
4. AGENDA
5. INSERT DESIGNATOR, IF NEEDED 5 Agenda • Why upstream? • QA in OpenStack • Creating a CI job and tests by example • Tests are merged and running! Am I done? • Final thoughts
6. WHY UPSTREAM?
7. INSERT DESIGNATOR, IF NEEDED 7 Why should we care about submitting our tests upstream? • Community support • If upstream, the tests are automated! • Features are verified early in the cycle Why upstream?
8. QA IN OPENSTACK
9. INSERT DESIGNATOR, IF NEEDED 9 QA in OpenStack • Let’s get familiar about some of the tools OpenStack provides for QA • Also… How the OpenStack CI works? ◦ Besides code reviews, every patch in OpenStack goes through the gate ▪ Check gate ▪ Merge gate
10. INSERT DESIGNATOR, IF NEEDED 10 CHECK GATE MERGE GATE REVIEWS
11. INSERT DESIGNATOR, IF NEEDED 11 The projects • Quick overview of some of the main projects related to QA in OpenStack ◦ DevStack ◦ Tempest ◦ Grenade ◦ Rally ◦ Zuul
12. INSERT DESIGNATOR, IF NEEDED 12 DevStack • DevStack is one of the most famous projects in OpenStack, for obvious reasons ◦ Almost everyone get started in OpenStack by playing with DevStack • It provides a set of scripts to easily bring up a complete - developer friendly - OpenStack cloud environment • There is also the DevStack Gate project, responsible to setup the virtual machines where the gate tests run • DevStack also offers a plugin feature where custom environments can be built
13. INSERT DESIGNATOR, IF NEEDED 13 Tempest
14. INSERT DESIGNATOR, IF NEEDED 14 Tempest • Validates OpenStack clouds via integration tests that use the components REST APIs • Also provides mechanisms to write “scenarios tests” ◦ Tests that have a complex path and might be related to more than one OpenStack service • Tempest offers a plugin interface, so you can use it as a lib to write tests in different repositories
15. INSERT DESIGNATOR, IF NEEDED 15 Grenade
16. INSERT DESIGNATOR, IF NEEDED 16 Grenade • Upgrade testing ◦ Not only SQL upgrades ◦ Also checks if bizarre events don’t happen ▪ Delete all servers ▪ Delete projects or users
17. INSERT DESIGNATOR, IF NEEDED 17 Rally
18. INSERT DESIGNATOR, IF NEEDED 18 Rally • Rally is a benchmark tool for OpenStack ◦ Objective is to test the cloud’s scale • Deploys the cloud, verifies (runs Tempest) and execute overload scenarios ◦ Can also use a already deployed cloud
19. INSERT DESIGNATOR, IF NEEDED 19 Zuul • Zuul is the project that handles the gate! ◦ Run tests for all changes pushed to Gerrit ◦ Figures out patches dependencies ▪ Even between different repositories ◦ Handles rechecks! • Zuul status page: http://status.openstack.org/zuul/
20. INSERT DESIGNATOR, IF NEEDED 20
21. INSERT DESIGNATOR, IF NEEDED 21 • Not a project, but rather a feature from OpenStack infra • Test a change in Gerrit using systems are not officially supported by OpenStack and vote in the change Third-Party jobs
22. CREATING A CI JOB AND TESTS BY EXAMPLE
23. INSERT DESIGNATOR, IF NEEDED 23 Motivation • We know that OpenStack offers the tools, how can we use them in our favor? • Let’s understand by following a real example
24. INSERT DESIGNATOR, IF NEEDED 24 Federated Identity tests • Understanding the tests requirements ◦ Custom environment: extra systems are needed besides what is provided by default in OpenStack ◦ Really Keystone specific: it is related on how Keystone authenticate its users ▪ For other OpenStack services everything remains the same after the authentication • A tempest plugin looks suitable here
25. INSERT DESIGNATOR, IF NEEDED 25 The steps Create a Tempest plugin for Keystone Create a non-voting gate job for the new plugin Add the first set of tests Change the job to be voting
26. INSERT DESIGNATOR, IF NEEDED 26 Creating a Tempest plugin • We used the tempest-plugin-cookiecutter • Insert a new directory in Keystone’s tree: keystone_tempest_plugin ◦ Just the base structure to write new tests ▪ https://review.openstack.org/#/c/297292/
27. INSERT DESIGNATOR, IF NEEDED 27 • To run the tests: ◦ tox -e all-plugin -- keystone • At this point, we don’t actually run any tests
28. INSERT DESIGNATOR, IF NEEDED 28 Creating a non-voting job • We already have a base structure for the tests ◦ Now we need to run them in the gate • Add a job to the new plugin ◦ New jobs are added to the openstack-infra/project-config project ◦ Non-voting so we can check if it works ◦ https://review.openstack.org/#/c/298696/
29. INSERT DESIGNATOR, IF NEEDED 29
30. INSERT DESIGNATOR, IF NEEDED 30
31. INSERT DESIGNATOR, IF NEEDED 31
32. INSERT DESIGNATOR, IF NEEDED 32 Adding the first set of tests • At this point, we have: ◦ A base structure to write the tests ◦ A CI job that is capable of running them • Let’s add the first set of tests
33. INSERT DESIGNATOR, IF NEEDED 33 Federated Identity API tests • Check the API first so we know they are working ◦ Identity Providers API ◦ Service Providers API ◦ Mappings API ◦ Protocols API • https://review.openstack.org/#/q/topic:federation_integration_tests
34. INSERT DESIGNATOR, IF NEEDED 34 Making the job voting • At this point, we have: ◦ Tests for the Federated Identity API being run on each Keystone patch ◦ A non-voting CI job that • Let’s make the job voting ◦ https://review.openstack.org/#/c/321890
35. INSERT DESIGNATOR, IF NEEDED 35 • Just remove -nv suffix • Also add it to the merge gate
36. INSERT DESIGNATOR, IF NEEDED 36 Next steps • The work to check the Federated Identity feature is not complete yet! • We still need a scenario test the checks the different types of authentication paths ◦ A custom environment is needed
37. INSERT DESIGNATOR, IF NEEDED 37 The DevStack plugin • A plugin that handles the installation of a custom environment • A new directory is inserted in Keystone’s tree ◦ https://review.openstack.org/#/c/320623/
38. INSERT DESIGNATOR, IF NEEDED 38 • To enable the plugin ◦ # local.conf ◦ enable_plugin keystone $KEYSTONE_REPO
39. INSERT DESIGNATOR, IF NEEDED 39 The scenario tests • The tests assume a ready environment • Uses the already tested APIs to prepare the tests setup • They go through the complete workflow to authenticate a user based in the Federated Identity feature • https://review.openstack.org/#/c/324769/
40. TESTS ARE MERGED AND RUNNING! AM I DONE?
41. INSERT DESIGNATOR, IF NEEDED 41
42. INSERT DESIGNATOR, IF NEEDED 42 “Manual only” tests • There are tests scenarios that can’t be automated ◦ For example: Horizon WebSSO in the Federated Identity feature ▪ The APIs and authentication paths have automated tests ▪ We still need to check if the feature works as expected in Horizon
43. INSERT DESIGNATOR, IF NEEDED 43 Custom environments challenges • Downstream tests are still required if… ◦ … your downstream product runs in a different environment ▪ Building tools ▪ Packages versions ▪ Operating system ◦ … your tests take too long to execute ▪ Doesn’t make sense to have a third-party job to run on every change upstream ▪ Usually, downstream products have fewer features so it is simpler to run this kind of tests ◦ … you have proprietary environments that might be private
44. FINAL THOUGHTS
45. INSERT DESIGNATOR, IF NEEDED 45 Final thoughts • Submit your QA tests upstream! • What OpenStack provides? How can I use? • I have my tests upstream, do I need to do anything else?
46. THANK YOU plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHatNews