Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Final Master Thesis Presentation

Final Master Thesis Presentation

Philipp Naderer

June 01, 2016

More Decks by Philipp Naderer

Other Decks in Research


  1. Non-Functional Testing in Cloud Environments Philipp Naderer-Puiu

  2. General Idea How well are existing applications supported by PaaS

    providers from a (non-)functional viewpoint? The focus is on migrating JEE applications to Google App Engine. Find methods and tools to test PaaS providers for • fast response times • elasticity and scalability. 2
  3. Related Work • Strong focus on IaaS providers ◦ CPU

    and memory tests ◦ EC2 vs. Google Compute Engine vs. … • Efficient VM provisioning • Comparison of various NoSQL data stores • Load tests for traditional software stacks ◦ LAMP / JEE web applications ◦ SQL databases in the background • Cloud services for scientific computing 3
  4. Examined Areas of (Non-)Functional Testing • Portability of Applications ◦

    Migrating JPA-based applications to App Engine • Load & Performance Tests ◦ Simulation of high load situations ◦ Spot bottlenecks in production environments ◦ Identify limits of the NoSQL Google Cloud Datastore • Scalability Tests ◦ How do Java applications scale on Google App Engine? 4
  5. Migration Experiment Moving JPA-based JEE Applications to Google App Engine

  6. JPA: From JEE to App Engine • JPA as common

    ground for relational data ◦ JPA is a persistence API for Java objects ◦ It spans a wide range of relational database features ◦ App Engine provides a DataNucleus plugin for the NoSQL Datastore service • Various aspects of JPA have been tested ◦ Entity mappings & table definitions ◦ Relationships between entities ◦ Collection mappings ◦ Transaction support & entity group management ◦ Queries 6
  7. What is the Google Datastore? Colossus Bigtable Megastore Datastore 7

  8. Datastore vs. Relational DB Entities Key Data 123123 1150 Max

    Mustermann M 123,92 null null 1 937489 1220 Jennifer Johnson F 92,10 null T-Shirt 2 ... ... ... ... ... ... ... ... Relational Database Table & Entities NoSQL Datastore Entities 8
  9. Datastore Transactions • Datastore’s scope of transaction is defined by

    the entity group membership ◦ This membership is determined by an entity’s key ◦ RDBMS allow arbitrary entities to be part of a transaction ◦ “Owned relationships” enforce both sides to be member of the same entity group • Every insert / update / delete of an entity is always part of an entity group transaction • Only 25 entity groups can be part of a transaction 9
  10. Using JPA on App Engine is painful and frustrating! The

    Objectify framework turned out to be an opinionated alternative.
  11. Performance Case Study Finding evidence for good scalability and elasticity

    of App Engine web applications
  12. Why a case study? • We can only find evidence

    for scalability in real world environments, not in labs / simulations ◦ App Engine is a proprietary technology ◦ Impossible to derive any model from the implementation ◦ We don’t have any information for the global state of the cloud environment • Benchmarks are a valid tool for case studies ◦ App Engine has no advanced monitoring for customers ◦ Only reliable way is to measure response times • Goal: form a strong chain of evidence 12
  13. Other empirical research methods • Simulations ◦ Only possible for

    open source stacks ◦ Often used for proof of concept implementation, e.g. to show efficiency of load balancing algorithms • Surveys ◦ Not possible if the customer base is not disclosed ◦ Hard to find a representative group • Analysing existing artifacts ◦ Google does only provide a development server ◦ Development server is not representative 13
  14. Research Questions Is JMeter an adequate load test tool for

    cloud-based applications? How does App Engine react on incoming traffic? Any effects on the Datastore performance? What are the consequences of the 1 write per entity group limit in real world deployments? 14
  15. Testing on Virtual Infrastructure NYC2 NYC3 SFO1 Docker 15

  16. Running a Load Test Step 2: JMeter HTTP-based test PHP

    Webserver Step 4: Download and Merge Results 16 Local Machine Step 1: Shell Scripts as Controllers Step 3: Store results as a CSV file
  17. Key Results The following slides only show a selection of

    the gained insights and collected results
  18. Throughput: Light vs. Heavy Entities • 7,5 minutes vs. 43

    minutes test runtime • 12x higher error rate for heavyweight • 50 ms per write vs. 180 ms per write! 18
  19. Creating large entity groups Group entities of the same entity

    group into a single transaction during creates or updates. 19
  20. Parallel Writes to single Entity Group: Datastore Contention Most updates

    took only 1 attempt, but some are running extremely long. 20
  21. Research Questions – Answers Is JMeter an adequate load test

    tool for cloud-based applications? Yes, JMeter … • performed well in a distributed setup • was easy to maintain and execute • generated sufficient load to stress the application and the Datastore service 21
  22. Research Questions – Answers How does App Engine react on

    incoming traffic? Any effects on the Datastore performance? • Auto-scales the application, no upper limit for instance count found • During instant high load some minor performance issues have been found • Datastore read performance “unlimited”, write performance depends on instances 22
  23. Research Questions – Answers What are the consequences of the

    1 write per entity group limit in real world deployments? • Not strictly enforced by App Engine • 90 writes per second at the peak • Some writes stucked in multiple attempts • Conclusion: Datastore tries to minimize the effects of contention for most of the users 23
  24. SUMMARY • JPA-based migrations are hard in practice ◦ Objectify

    framework is a specialized alternative • Performance Case Study showed no bottlenecks ◦ Read performance is limited by the budget ◦ Write performance is not great, but avoidable by design • App Engine is a very specialized product ◦ Hard to migrate back to a on-premise stack ◦ Google’s Cloud Datastore is different to other NoSQL stores 24
  25. Merci! Photos courtesy of Google/Connie Zhou