Systematic and recomputable comparison of multi-cloud management platforms

With the growth and evolution of cloud applications, more and more architectures use hybrid cloud bindings to optimally use virtual resources regarding pricing policies and performance. This process has led to the creation of multi-cloud management platforms as well as abstraction libraries. At the moment, many (multi-)cloud management platforms (CMPs) are designed to cover the functional requirements. Along with growing adoption and industrial impact of such solutions, there is a need for a comparison and test environment which automatically assesses and compares existing platforms and helps in choosing the optimal one. This paper focuses on the creation of a suitable testbed concept and an actual extensible software prototype which makes multi-cloud experiments repeatable and reusable by other researchers. The work is evaluated by an exemplary comparison of 4 CMPs bound to AWS, showcasing standardised output formats and evaluation criteria.

Transcript

1. Systematic and recomputable comparison of
   multi-cloud management platforms
   Oleksii Serhiienko, Josef Spillner
   10th IEEE CloudCom Cyprus December 2018
   Service Prototyping Lab @
   Zurich University of Applied Sciences
   Switzerland
   

   View Slide

2. Cloud management platform (CMP)
   - Growing needs of multi cloud application and hybrid cloud
   → CMPs gained much popularity
   - Many solutions supporting different needs and various of platforms
   - CMPs can be:
   - Standalone platform
   - Multi-cloud api library
   - Website based
   

   View Slide

3. CMP evaluation
   - Evaluation criteria:
   - Time, sec
   - Memory consumption, KB
   - CPU, sec
   - One of each type is taken to make proof of concept
   - Flexible software:
   - Easy to add new platforms
   - Easy to add new evaluation criteria
   

   View Slide

4. Related work
   - CloudCom 2015: An Empirical Study for Evaluating the Performance of
   Jclouds
   - Measuring Jclouds multi-cloud tool kit and compare to native AWS library
   - Download/upload file
   - Сritical evaluation on Jclouds and Cloudify abstract APIs against EC2,
   Azure and HP-Cloud
   - Create a prototype tool that will evaluate Jclouds andCloudify
   

   View Slide

5. Requirements and approach
   - CoMParable CMPs (CMP²) requirements:
   - Comfort
   - Statistical correctness
   - Reproducibility
   - Extensibility
   - Set of decorators:
   - Timing
   - Docker consumption
   - Python consumption
   - Tagging
   

   View Slide

6. Platforms choice
   - Web Platforms:
   - CloudcheckR
   - Libraries:
   - Libcloud
   - Containers:
   - Composed Containers:
   - MistIO
   - Single Container:
   - ManageIQ
   

   View Slide

7. Input data
   

   View Slide

8. Architecture
   

   View Slide

9. Experimental setup
   - Hardware
   - RAM: 4 GB
   - VCPUs: 2 vCPUs clocked at 2500 Mhz
   - Disk: 40 GB
   - OS: Ubuntu 16.04.4 LTS
   - Software:
   - Mist.io: Cloud Management Platform version: 2.0
   - ManageIQ: gaprindashvili-3
   - CloudcheckR: last update May 21, 2018
   - Apache Libcloud: version 2.3.0
   

   View Slide

10. Results
    - During this work was developed testbed for CMPs for recomputable
    experiments
    - Easy to extend
    - Generates graphs and latex table
    

    View Slide

11. Compare results with related work
    

    View Slide

12. “What happened since camera-ready submission“
    - Сloudbridge (library)
    - Cloudify (docker based)
    - Open Source released
    - Try it out!
    

    View Slide

13. Conclusion
    - Test environment and an architecture for multi-platform testing were
    created
    - The architecture is modular and very flexible which provides the
    possibility of its low-effort expansion.
    - All the data and findings are published as open source
    - Open data to keep the study reusable and repeatable
    one more thing...
    

    View Slide

14. One more thing...
    Last-minute
    registration still
    possible…
    140+ talks
    190+ attendees
    11th IEEE/ACM International Conference on Utility and Cloud Computing
    Zurich | CH | Dec 17-20, 2018
    http://www.ucc-conference.org/
    

    View Slide