Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Final Master Thesis Presentation

Final Master Thesis Presentation

Philipp Naderer

June 01, 2016
Tweet

More Decks by Philipp Naderer

Other Decks in Research

Transcript

  1. Non-Functional Testing in
    Cloud Environments
    Philipp Naderer-Puiu

    View full-size slide

  2. General Idea
    How well are existing applications supported by
    PaaS providers from a (non-)functional viewpoint?
    The focus is on migrating JEE applications to
    Google App Engine.
    Find methods and tools to test PaaS providers for
    ● fast response times
    ● elasticity and scalability.
    2

    View full-size slide

  3. Related Work
    ● Strong focus on IaaS providers
    ○ CPU and memory tests
    ○ EC2 vs. Google Compute Engine vs. …
    ● Efficient VM provisioning
    ● Comparison of various NoSQL data stores
    ● Load tests for traditional software stacks
    ○ LAMP / JEE web applications
    ○ SQL databases in the background
    ● Cloud services for scientific computing
    3

    View full-size slide

  4. Examined Areas of
    (Non-)Functional Testing
    ● Portability of Applications
    ○ Migrating JPA-based applications to App Engine
    ● Load & Performance Tests
    ○ Simulation of high load situations
    ○ Spot bottlenecks in production environments
    ○ Identify limits of the NoSQL Google Cloud Datastore
    ● Scalability Tests
    ○ How do Java applications scale on Google App Engine?
    4

    View full-size slide

  5. Migration Experiment
    Moving JPA-based JEE Applications
    to Google App Engine

    View full-size slide

  6. JPA: From JEE to App Engine
    ● JPA as common ground for relational data
    ○ JPA is a persistence API for Java objects
    ○ It spans a wide range of relational database features
    ○ App Engine provides a DataNucleus plugin
    for the NoSQL Datastore service
    ● Various aspects of JPA have been tested
    ○ Entity mappings & table definitions
    ○ Relationships between entities
    ○ Collection mappings
    ○ Transaction support & entity group management
    ○ Queries 6

    View full-size slide

  7. What is the Google Datastore?
    Colossus
    Bigtable
    Megastore
    Datastore
    7

    View full-size slide

  8. Datastore vs. Relational DB Entities
    Key Data
    123123 1150 Max Mustermann M 123,92 null null 1
    937489 1220 Jennifer Johnson F 92,10 null T-Shirt 2
    ... ... ... ... ... ... ... ...
    Relational Database Table & Entities
    NoSQL Datastore Entities
    8

    View full-size slide

  9. Datastore Transactions
    ● Datastore’s scope of transaction is defined
    by the entity group membership
    ○ This membership is determined by an entity’s key
    ○ RDBMS allow arbitrary entities to be part of a transaction
    ○ “Owned relationships” enforce both sides
    to be member of the same entity group
    ● Every insert / update / delete of an entity
    is always part of an entity group transaction
    ● Only 25 entity groups can be part of a transaction
    9

    View full-size slide

  10. Using JPA on App Engine
    is painful and frustrating!
    The Objectify framework turned out
    to be an opinionated alternative.

    View full-size slide

  11. Performance Case Study
    Finding evidence for good scalability and
    elasticity of App Engine web applications

    View full-size slide

  12. Why a case study?
    ● We can only find evidence for scalability in
    real world environments, not in labs / simulations
    ○ App Engine is a proprietary technology
    ○ Impossible to derive any model from the implementation
    ○ We don’t have any information for the global state
    of the cloud environment
    ● Benchmarks are a valid tool for case studies
    ○ App Engine has no advanced monitoring for customers
    ○ Only reliable way is to measure response times
    ● Goal: form a strong chain of evidence
    12

    View full-size slide

  13. Other empirical research methods
    ● Simulations
    ○ Only possible for open source stacks
    ○ Often used for proof of concept implementation,
    e.g. to show efficiency of load balancing algorithms
    ● Surveys
    ○ Not possible if the customer base is not disclosed
    ○ Hard to find a representative group
    ● Analysing existing artifacts
    ○ Google does only provide a development server
    ○ Development server is not representative
    13

    View full-size slide

  14. Research Questions
    Is JMeter an adequate load test tool
    for cloud-based applications?
    How does App Engine react on incoming traffic?
    Any effects on the Datastore performance?
    What are the consequences of the 1 write per
    entity group limit in real world deployments?
    14

    View full-size slide

  15. Testing on Virtual Infrastructure
    NYC2
    NYC3
    SFO1
    Docker
    15

    View full-size slide

  16. Running a Load Test
    Step 2:
    JMeter HTTP-based test
    PHP
    Webserver
    Step 4:
    Download and Merge Results
    16
    Local Machine
    Step 1: Shell Scripts as Controllers
    Step 3:
    Store results as a CSV file

    View full-size slide

  17. Key Results
    The following slides only show a selection
    of the gained insights and collected results

    View full-size slide

  18. Throughput: Light vs. Heavy Entities
    ● 7,5 minutes vs. 43 minutes test runtime
    ● 12x higher error rate for heavyweight
    ● 50 ms per write vs. 180 ms per write!
    18

    View full-size slide

  19. Creating large entity groups
    Group entities of the same entity group
    into a single transaction during creates or updates.
    19

    View full-size slide

  20. Parallel Writes to single Entity Group:
    Datastore Contention
    Most updates took only 1 attempt,
    but some are running extremely long.
    20

    View full-size slide

  21. Research Questions – Answers
    Is JMeter an adequate load test tool
    for cloud-based applications?
    Yes, JMeter …
    ● performed well in a distributed setup
    ● was easy to maintain and execute
    ● generated sufficient load to stress
    the application and the Datastore service
    21

    View full-size slide

  22. Research Questions – Answers
    How does App Engine react on incoming traffic?
    Any effects on the Datastore performance?
    ● Auto-scales the application,
    no upper limit for instance count found
    ● During instant high load some minor
    performance issues have been found
    ● Datastore read performance “unlimited”,
    write performance depends on instances
    22

    View full-size slide

  23. Research Questions – Answers
    What are the consequences of the 1 write per
    entity group limit in real world deployments?
    ● Not strictly enforced by App Engine
    ● 90 writes per second at the peak
    ● Some writes stucked in multiple attempts
    ● Conclusion: Datastore tries to minimize the
    effects of contention for most of the users
    23

    View full-size slide

  24. SUMMARY
    ● JPA-based migrations are hard in practice
    ○ Objectify framework is a specialized alternative
    ● Performance Case Study showed no bottlenecks
    ○ Read performance is limited by the budget
    ○ Write performance is not great, but avoidable by design
    ● App Engine is a very specialized product
    ○ Hard to migrate back to a on-premise stack
    ○ Google’s Cloud Datastore is different to other NoSQL stores
    24

    View full-size slide

  25. Merci!
    Photos courtesy of Google/Connie Zhou

    View full-size slide