Making Cloudy Peanut Butter Cups: Apache CloudStack + Riak CS

Building complete private cloud architectures that support both traditional application virtualization and emerging distributed cloud systems requires compute and object storage storages. Together Apache CloudStack and Basho's Riak CS provide the infrastructure to support both of workloads. The presentation covers the following topics:

1. Why Private Cloud?
2. Anatomy of a Private Cloud
3. Building a Apache CloudStack Compute Offering
4. Large Object Storage using Riak CS
5. Your Own Private Cloud: The Riak CS Apache CloudStack Integration Roadmap

This presentation was originally delivered to the Silicon Valley Riak group on 27 June 2013.

Transcript

1. MAKING CLOUDY PEANUT BUTTER CUPS: APACHE CLOUDSTACK + RIAK CS John Burwell (jburwell@basho.com | @john_burwell) Thursday, June 27, 13
2. Who Am I • Consulting Engineer @ Basho Technologies • Apache CloudStack PMC Member • Ran operations and designed automated provisioning for hybrid analytic/virtualization clouds • Led architectural design and server-side development of a SaaS physical security platform Thursday, June 27, 13
3. Thursday, June 27, 13
4. Hopefully not ... Thursday, June 27, 13
5. Thursday, June 27, 13
6. Thursday, June 27, 13
7. Thursday, June 27, 13
8. Composable Infrastructure • Resource Pooling • Rapid Elasticity • Self Service • Metering/Charge Back Thursday, June 27, 13
9. Doesn’t vSphere/ XCP/XenServer already do that? Thursday, June 27, 13
10. Traditional Workload Management Network Hypervisor Cluster Hypervisor Cluster Shared
11. Cloud Workload • Reliable • Fault Tolerant • Commodity Hardware • Self-Service provisioning • Dynamic scale out model SDN Rack Rack Object
12. Traditional Workload Cloud Workload The Chicken Or The Egg? Thursday, June 27, 13
13. Got It ... Why Go Private? • Leverage existing infrastructure investment • Customization and Control • Regulatory/Security Requirements • Cost Thursday, June 27, 13
14. Open Source Private Cloud + Compute Services Object Storage Services Thursday, June 27, 13
15. Make infrastructure boring. Thursday, June 27, 13
16. What Is Apache CloudStack? §Open source software that deploys and manages large networks of virtual machines, as a highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform. §CloudStack is a Top-Level Project at the Apache Software Foundation. Thursday, June 27, 13
17. Apache CloudStack Users Thursday, June 27, 13
18. Infrastructure Model • Management Server Farm • Management and provisioning tasks • Zone • Collection of pods, network offerings and secondary storage • Pod • Collection of clusters in the same failure boundary • Cluster • A grouping of hosts and their associated storage • Hosts • Servers onto which services will be provisioned • Primary Storage • VM disk storage • Network • Logical network associated with service offerings • Secondary Storage • Template, snapshot and ISO storage Zone CloudStack Pod Cluster Host Host Network Primary Storage VM VM CloudStack Pod Cluster Secondary Storage Thursday, June 27, 13
19. Management Server Farm • MS is stateless. MS can be deployed as physical server or VM • Single MS node can manage up to 10K hosts. Multiple nodes can be deployed for scale or redundancy • RHEL/CentOS (5.4+), Ubuntu (10.0.4+), Fedora (16+) User
20. Got Scale? • Running in production at >30,000 physical hosts, supported by only 4 management server instances • Even greater scale by using CloudStack in a Regional model • Remember: The separation of control, management and data planes is critical for cloud platform scale. Thursday, June 27, 13
21. Deployment Architecture • Hypervisor is the basic unit of scale. • Cluster consists of one ore more hosts of same hypervisor • All hosts in cluster have access to shared (primary) storage • Pod is one or more clusters, usually with L2 switches. • Availability Zone has one or more pods, has access to secondary storage. • One or more zones represent a cloud                                                                                                           Pod  1 …. Cluster  N L2 Host 2 Cluster  1 Host 1 Primary Storage Zone  1 …. L3 Secondary Storage Pod  N Management Server Farm Internet Thursday, June 27, 13
22. Hypervisors • Xen / XCP (Ubuntu 12.04.2) • XenServer (6.1) • KVM (Ubuntu 12.04.2 / CentOS 6) • VMWare vCenter (4.1 / 5.1) Thursday, June 27, 13
23. Storage • ISCSI • NFS • Local Filesystem Storage • Ceph • S3-backed Secondary Storage • Swift-backed Secondary Storage Thursday, June 27, 13
24. Networking • VLAN and Security Group isolation • Load Balancer Support • Juniper SRX, F5, Nicira NVP and Open vSwitch integration Thursday, June 27, 13
25. 4.1: Hot Off The Presses • AWS style regions • Event pub-sub framework (RabbitMQ implementation) • Advanced search within the UI • API Server request throttling • API Discoverer Service • Self-Service Password Reset • Users directly changing their API keys • EC2 query API • Cloudmonkey CLI • Autoscale (F5-specific) Much, much more .... Thursday, June 27, 13
26. 4.2: On-Deck • Midonet and BigSwitch SDN integrations • Enhanced Baremetal provisioning • VM I/O Throttling • Hyper-V 2012 Support • LXC Support • Cisco VSG integration • Cisco ASA 1000V • VM Affinity Rules • VM Affinity Rules • Zone-wide primary storage • Security group isolation in Advanced Network zones • Dedicating resources to domains and accounts • IP Address reservation w/o a vNIC allocation • vSphere DVS support • UI Plugin framework Thursday, June 27, 13
27. First A Little Riak • Dynamo-inspired, masterless key-value store • Apache License • Always available for writes • Highly available reads • Secondary Indexes/Search • MapReduce • Operations Friendly Thursday, June 27, 13
28. ... Then A Little Riak CS • Built on top Riak • Apache Licensed • S3-compatible API • Large object support and multipart upload • Multi-tenancy and per-user reporting • Operations Friendly Thursday, June 27, 13
29. How It Works Large
30. Riak CS Roadmap • Keystone integration • OpenStack Storage API v1 (aka Swift) support • Performance improvements around large bucket operations Thursday, June 27, 13
31. The Complete Private Cloud Management Network Hypervisor Cluster Hypervisor Cluster Shared
32. Apache CloudStack + Riak CS Roadmap • Now • S3-backed Secondary Storage • 4.2 • Partial support for using object stores as secondary storage • 4.3 • Integrated authentication • Complete support for using object stores as secondary storage Thursday, June 27, 13
33. Join Our Community • Learn more @ http://cloudstack.apache.org • Grab it @ http://cloudstack.apache.org/downloads.html • Follow us @cloudstack • Help from the cloudstack-users@cloudstack.apache.org mailing list or on Freenode @ #cloudstack • Contribute on cloudstack-dev@cloudstack.apache.org mailing list or on Freenode @ #cloudstack-dev Thursday, June 27, 13
34. Up And Running With Riak CS • Learn more @ http://basho.com/riak-cloud-storage • Grab a distribution @ http://docs.basho.com/riakcs/latest/ riakcs-downloads/ • Follow us @basho • Help from the riak-users@lists.basho.com mailing list or on FreeNode @ #riak • Contribute @ http://github.com/basho/riak_cs Thursday, June 27, 13
35. Thoughts? Questions? Thursday, June 27, 13
36. Thank you! Slides available @ http://speakerdeck.com/jburwell Thursday, June 27, 13