Final Master Thesis Presentation

Transcript

1. Non-Functional Testing in Cloud Environments Philipp Naderer-Puiu
2. General Idea How well are existing applications supported by PaaS providers from a (non-)functional viewpoint? The focus is on migrating JEE applications to Google App Engine. Find methods and tools to test PaaS providers for • fast response times • elasticity and scalability. 2
3. Related Work • Strong focus on IaaS providers ◦ CPU and memory tests ◦ EC2 vs. Google Compute Engine vs. … • Efficient VM provisioning • Comparison of various NoSQL data stores • Load tests for traditional software stacks ◦ LAMP / JEE web applications ◦ SQL databases in the background • Cloud services for scientific computing 3
4. Examined Areas of (Non-)Functional Testing • Portability of Applications ◦ Migrating JPA-based applications to App Engine • Load & Performance Tests ◦ Simulation of high load situations ◦ Spot bottlenecks in production environments ◦ Identify limits of the NoSQL Google Cloud Datastore • Scalability Tests ◦ How do Java applications scale on Google App Engine? 4
5. Migration Experiment Moving JPA-based JEE Applications to Google App Engine
6. JPA: From JEE to App Engine • JPA as common ground for relational data ◦ JPA is a persistence API for Java objects ◦ It spans a wide range of relational database features ◦ App Engine provides a DataNucleus plugin for the NoSQL Datastore service • Various aspects of JPA have been tested ◦ Entity mappings & table definitions ◦ Relationships between entities ◦ Collection mappings ◦ Transaction support & entity group management ◦ Queries 6
7. What is the Google Datastore? Colossus Bigtable Megastore Datastore 7
8. Datastore vs. Relational DB Entities Key Data 123123 1150 Max Mustermann M 123,92 null null 1 937489 1220 Jennifer Johnson F 92,10 null T-Shirt 2 ... ... ... ... ... ... ... ... Relational Database Table & Entities NoSQL Datastore Entities 8
9. Datastore Transactions • Datastore’s scope of transaction is defined by the entity group membership ◦ This membership is determined by an entity’s key ◦ RDBMS allow arbitrary entities to be part of a transaction ◦ “Owned relationships” enforce both sides to be member of the same entity group • Every insert / update / delete of an entity is always part of an entity group transaction • Only 25 entity groups can be part of a transaction 9
10. Using JPA on App Engine is painful and frustrating! The Objectify framework turned out to be an opinionated alternative.
11. Performance Case Study Finding evidence for good scalability and elasticity of App Engine web applications
12. Why a case study? • We can only find evidence for scalability in real world environments, not in labs / simulations ◦ App Engine is a proprietary technology ◦ Impossible to derive any model from the implementation ◦ We don’t have any information for the global state of the cloud environment • Benchmarks are a valid tool for case studies ◦ App Engine has no advanced monitoring for customers ◦ Only reliable way is to measure response times • Goal: form a strong chain of evidence 12
13. Other empirical research methods • Simulations ◦ Only possible for open source stacks ◦ Often used for proof of concept implementation, e.g. to show efficiency of load balancing algorithms • Surveys ◦ Not possible if the customer base is not disclosed ◦ Hard to find a representative group • Analysing existing artifacts ◦ Google does only provide a development server ◦ Development server is not representative 13
14. Research Questions Is JMeter an adequate load test tool for cloud-based applications? How does App Engine react on incoming traffic? Any effects on the Datastore performance? What are the consequences of the 1 write per entity group limit in real world deployments? 14
15. Testing on Virtual Infrastructure NYC2 NYC3 SFO1 Docker 15
16. Running a Load Test Step 2: JMeter HTTP-based test PHP Webserver Step 4: Download and Merge Results 16 Local Machine Step 1: Shell Scripts as Controllers Step 3: Store results as a CSV file
17. Key Results The following slides only show a selection of the gained insights and collected results
18. Throughput: Light vs. Heavy Entities • 7,5 minutes vs. 43 minutes test runtime • 12x higher error rate for heavyweight • 50 ms per write vs. 180 ms per write! 18
19. Creating large entity groups Group entities of the same entity group into a single transaction during creates or updates. 19
20. Parallel Writes to single Entity Group: Datastore Contention Most updates took only 1 attempt, but some are running extremely long. 20
21. Research Questions – Answers Is JMeter an adequate load test tool for cloud-based applications? Yes, JMeter … • performed well in a distributed setup • was easy to maintain and execute • generated sufficient load to stress the application and the Datastore service 21
22. Research Questions – Answers How does App Engine react on incoming traffic? Any effects on the Datastore performance? • Auto-scales the application, no upper limit for instance count found • During instant high load some minor performance issues have been found • Datastore read performance “unlimited”, write performance depends on instances 22
23. Research Questions – Answers What are the consequences of the 1 write per entity group limit in real world deployments? • Not strictly enforced by App Engine • 90 writes per second at the peak • Some writes stucked in multiple attempts • Conclusion: Datastore tries to minimize the effects of contention for most of the users 23
24. SUMMARY • JPA-based migrations are hard in practice ◦ Objectify framework is a specialized alternative • Performance Case Study showed no bottlenecks ◦ Read performance is limited by the budget ◦ Write performance is not great, but avoidable by design • App Engine is a very specialized product ◦ Hard to migrate back to a on-premise stack ◦ Google’s Cloud Datastore is different to other NoSQL stores 24
25. Merci! Photos courtesy of Google/Connie Zhou