How SAP is using Python to test its database SAP HANA

SAP operates one of the largest test infrastructure to test its in-memory database SAP HANA. The infrastructure provides different services like continues integration, code coverage and code linting for a huge C++ project with Python test coding. These services are essential for the development teams and quality specialists. Without these services developing and shipping of new SAP HANA version wouldn’t be possible. In 2010, we started with a single Jenkins master with ten nodes. But to keep our testing time acceptable for the growing number of developers we had to scale up and that led to multiple different scaling challenges. The current test infrastructure is powered by more than thousand physical servers. Scaling of the infrastructure was only possible with custom optimizations like improved scheduling, expressive test configuration and robust tooling implemented in our favorite language Python. With the flexibility and power of Python it’s possible for developers to implement complex test scenarios to verify features and mitigate regressions. On infrastructure side, it has been easier to extend, optimize and adapt the infrastructure for new requirements like different CPU architectures and newer Operating systems versions.

This talk provides insights and stories how we scaled and improved our test infrastructure and how new technologies like Linux Containers can improve automated testing and software quality assurance.

Transcript

1. Christoph Heer EuroPython 2017 - July 11 How SAP is using Python to test its database SAP HANA
2. 2 PUBLIC © 2017 SAP SE or an SAP affiliate company. • In-Memory database • Column & Row store • OLAP + OLTP • Single host or distributed • SQL • C++ & Python SAP HANA
3. 3 PUBLIC © 2017 SAP SE or an SAP affiliate company. Python + SAP HANA SAP HANA Python Client DBAPI SQLDBC app.py SQLAlchemy Django
4. 4 PUBLIC © 2017 SAP SE or an SAP affiliate company. Testing a database Unit/Component tests C++ Integration tests Python End-to-end tests Python
5. 5 PUBLIC © 2017 SAP SE or an SAP affiliate company. Gerrit Pre-merge Quality Assurance Central Repository Pending changes Build Test Static code analysis Style checks C++ sanitizers Review
6. 6 PUBLIC © 2017 SAP SE or an SAP affiliate company. Implementation - 2010 Gerrit master Jenkins Gerrit Trigger Node 1 | | | | | | | | | | Job Queue Node 2 Node 3 Node n Configuration Build Setup Test Pull QA Database Web UI …
7. 7 PUBLIC © 2017 SAP SE or an SAP affiliate company. 600 Developers 1,300 Jenkins nodes 408 TB Memory 700 Commits per day 36,000h Test runtime per day 13 mil. lines Python tests
8. 8 PUBLIC © 2017 SAP SE or an SAP affiliate company. Scaling domains Test Runtime Artifacts Healthy Test Environment Test Scheduling
9. 9 PUBLIC © 2017 SAP SE or an SAP affiliate company. Test Runtime: Divide and conquer test Build Setup Test Setup Store Test Build Pull Pull Store Test Build Pull Test Test hours 1 2 3 4 5 6 7 Setup Test Ready for review Setup Setup Setup
10. 10 PUBLIC © 2017 SAP SE or an SAP affiliate company. • Regression, sporadic failure or infrastructure problem • Rerun test to • confirm stable failure => regression • exclude infrastructure problem • Who restarts failed tests? Test Scheduling: Handle test failures git 003adae TestBaseFunctionality FAILED git 003adae TestBaseFunctionality_rerun OK
11. 11 PUBLIC © 2017 SAP SE or an SAP affiliate company. Configuration: • What tests should run? • Branch specific configuration • Layered testing • Exclude current broken tests Observe and react: • Re-schedule failed tests • Automated review • Notify about completion Test Scheduling Database Jenkins Waiter Build Config UI External service e.g. Bugzilla
12. 12 PUBLIC © 2017 SAP SE or an SAP affiliate company. • Nightly test runs should be complete next morning • Test bug fixes before new features • Finish testing of 95% tested commit Test Scheduling: Queuing Feature git c91abba – Test 1 Feature git c91abba – Test 2 Feature git c91abba – Test 3 Restart git 003adae – Test 3 nightly git 412fdae – Test 1 nightly git 412fdae – Test 2 nightly git 412fdae – Test 3 Waiter Jenkins: First in – first out Node ? Bugfix git c91abba – Test 2
13. 13 PUBLIC © 2017 SAP SE or an SAP affiliate company. Prioritized Test Queue Test Scheduling: Queuing Waiter nightly git 412fdae – Test 1 nightly git 412fdae – Test 2 nightly git 412fdae – Test 3 100 100 100 Bugfix git afa25ce – Test 2 50 Restart git 003adae – Test 3 70 Feature git c91abba – Test 1 Feature git c91abba – Test 2 Feature git c91abba – Test 3 30 30 30 Jenkins 2 Queue processor Jenkins 1 Jenkins n
14. 14 PUBLIC © 2017 SAP SE or an SAP affiliate company. Installer binaries (15 GB) Test data (2MB – 800GB) Artifacts NFS share NFS share Setup Fetch Test Local cache Local cache NFS share NFS share 9PB data transfer per week hitting bandwidth limits 66% less traffic to 3PB per week
15. 15 PUBLIC © 2017 SAP SE or an SAP affiliate company. • External dependencies • will fail • Parallel testing on host • Noisy neighbors • Health check before and while testing • Failure marks test as invalid • Reschedule • Invalid tests are invisible for developers Healthy Test Environment Setup Fetch Test Check
16. 16 PUBLIC © 2017 SAP SE or an SAP affiliate company. Implementation – 2017 Gerrit master Jenkins n Node 1 | | | | | | | | | | Job Queue Node 2 Node 3 Node n T T T T Test T T T T T T T T T T T T QA Database Web UI Queue processor Setup Fetch Check Waiter Build Infrastructure QA Database …
17. 17 PUBLIC © 2017 SAP SE or an SAP affiliate company. Learning curve Non-developers can write tests Development velocity idea to production <1 day Community virtualenv, pip, Fabric, Sentry etc. Platform independent CPU architecture + OS Python
18. 18 PUBLIC © 2017 SAP SE or an SAP affiliate company. • Scale for ~3000 nodes • Resource based scheduling with Apache Mesos • Isolation with Linux Containers • Guaranteed resources: Memory, CPU, IO • Python 3 • asyncio, type annotations • even better standard library Outlook
19. Christoph Heer christoph.heer@sap.com @ChristophHeer Thank you. We are hiring! Open Python positions (Junior) Python Developer: https://jobs.sap.com/s/kEui5u (Junior) Quality Engineer: https://jobs.sap.com/s/lKieYY (Senior) Quality Engineer: https://jobs.sap.com/s/2Cz9B8 Linux System Administrator: https://jobs.sap.com/s/5uvE3u
20. 20 PUBLIC © 2017 SAP SE or an SAP affiliate company. This presentation is licensed to the public under Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0). The information contained herein may be changed without prior notice. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation, and SAP SE’s or its affiliated companies’ strategy and possible future developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, and they should not be relied upon in making purchasing decisions. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. All other product and service names mentioned are the trademarks of their respective companies. See http://global.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices. © 2017 SAP SE or an SAP affiliate company.