Self Heal Your OpenStack Control Plane!

CloudPerceptions (Shixiong Shang and Randy Tuttle) and our friend Derek Chamorro from eBay delivered "Self Heal Openstack Control Plane" speaking session in OpenStack Summit Austin on April 26, 2016.

The video recording can be found at: https://youtu.be/lkhkY_52vJk
The demo video clips are posted to: https://www.youtube.com/channel/UCKeiS_ciRfweultTNNlNjIw
The corresponding BLOG can be found at http://blog.cloudperceptions.com and http://therandomsecurityguy.com

Transcript

1. Self Heal Your OpenStack Control Plane! Derek Chamorro (eBay) Shixiong Shang (CloudPerceptions) Randy Tuttle (CloudPerceptions) * OpenStack is a trademark of OpenStack Foundation
2. Randy Tuttle Co-Founder and CTO Cloud Perceptions email: rtuttle@cloudperceptions.com SP VoIP, SP Video OpenStacker since Folsom Release Triangle Kubernete Meetup Organizer Introduction Derek Chamorro Cloud Security Architect eBay Twitter: @theredinthesky email: dchamorro@ebay.com https://therandomsecurityguy.com OpenStacker since Folsom Release Kubernetes Nerd Shixiong Shang Founder and CEO CloudPerceptions email: shshang@cloudperceptions.com CCIE R&S, SP Video OpenStacker since Folsom Release OpenStack Vancouver Speaker Triangle Kubernete Meetup Organizer
3. Agenda • Our Vision • OpenStack Pain Points • Kubernetes Overview • Proposed Solution and Benefits • Demo • Summary • Conclusion • Next Steps • Q/A 3
4. Our Vision (or what would be nirvana) Improved SLA Simple to scale Self-healing Consistent and Repeatable Quickly Operationalized Readily Customized 4 Easily Deployed
5. Deployment - "We need it fast, and we need it yesterday!!". [Some] OpenStack Pain Points Inconsistencies in build and deployment environments - "Say what!!!!? We didn't know you needed version x.0.0 vs. version x.1.1!!" Operationalization - "Wait, my service's latency is how high?!!! Why did I not know that?!!" 5 Easily Deployed Quickly Operationalized Consistent and Repeatable Self-healing
6. Control Plane Design Pain Points 6 Significant Investment in Pre-deployment Planning and Engineering Inadequate Post-deployment Monitoring Lack of service healing Lack of elasticity after service provisioning
7. OpenStack Control Plane High Availability 7 rabbitMQ mysql/ mariaDB keystone nova api/ scheduler/ conductor glance api/ registry cinder api/ scheduler neutron- server swift heat-api rabbitMQ mysql/ mariaDB keystone nova api/ scheduler/ conductor glance api/ registry cinder api/ scheduler neutron- server swift heat-api HAProxy HAProxy AIO Compute Compute nova api/ scheduler/ conductor 2 vCPU - 4Gb RAM x 2 nova api/ scheduler/ conductor mysql rabbitMQ $ netstat -ant | grep 8774.*LISTEN $ service nova-scheduler status $ tail -100 /var/log/nova/scheduler.log rabbitMQ mysql/ mariaDB keystone nova api/ scheduler/ conductor glance api/registry cinder api/ scheduler neutron- server swift heat-api rabbitMQ mysql/ mariaDB keystone nova api/ scheduler/ conductor glance api/registry cinder api/ scheduler neutron- server swift heat-api HAProxy HAProxy AIO Compute Compute
8. Kubernetes 8
9. Kubernetes Overview (and our perceived benefits) 9 1. What is it? 2. What is the advantage? 3. Why do we think it will benefit OpenStack community? “Kubernetes is an open source automation framework for deploying, managing, and scaling applications via Docker across clusters of hosts.” • Declarative • Scale • Easy to Build/Update • Efficient • Self-healing • Improved SLA’s • Consistent
10. Worker Node Master Node Kubernetes Architecture 10 Scheduling REST Services Scheduler etcd Control mgr API Server kube-proxy kubelet container container Docker Pod Pod
11. Kubernetes OpenStack Control Plane Architecture Kube Master Kube Proxy Docker Worker Node Kube Proxy rabbitMQ Pod nova-api Pod glance-api Pod Docker neutron- server Pod keystone Pod mysql Pod Worker Node 11 rabbitMQ mysql/ mariaDB keystone nova api/ scheduler/ conductor glance api/registry cinder api/schedul er neutron- server swift heat-api rabbitMQ mysql/ mariaDB keystone nova api/ scheduler/ conductor glance api/registry cinder api/schedul er neutron- server swift heat-api HAProxy HAProxy AIO Compute Compute
12. glance-api-rc- qyvvp Proposed Solution (Self Healed OpenStack Control Plane) 12 glance-api Kube Proxy TCP 9292 kubelet httpGet path: / port: 9292 glance-api glance-api-rc- r1gdl glance-api X
13. Why We Selected Kubernetes? 13 Provisioning and Orchestration CI/CD Consistency Scaling Self-healing
14. 14 Demo: Self-healing OpenStack Control Plane Plot Summary (Trilogy): OpenStack Deployment and Verification Self-healing! Operational Capability Credits: Actors: One Master/Worker VM, Two Worker VMs (172.16.7.11 - .13) Screenwriters: Ubuntu 14.04.3 Director: OpenStack Liberty Community Distribution, Kubernetes 1.2, Docker 1.10.3 OpenStack Austin Summit Special Edition Live Demo is also available!
15. Summary 15 Easily Deployed Quickly Operationalized Consistent and Repeatable Self-healing V.S.
16. Conclusion 16 CI/CD is quite feasible Operations will become more challenging… It is all about improving OpenStack deployment can be easy and fast but still achievable efficiency
17. Next Steps: __? 17 Make It Better Scalable, Flexible, Reliable, Novel,
18. 18 Q/A
19. 19 See You Soon!