Share
Embed
Exoscale: Operating a public cloud
This is the talk I gave at the CloudStack Collaboration Conference Europe 2013 held in Amsterdam.
It focuses on how we operate our Cloudstack based "Open Cloud" offering and an overview of the complete ecosystem we've built around it.
exoscale
November 22, 2013
Transcript
31 slides
-
#whoami ! Loic Lambiel – Email : [email protected]
! Passionate sysadmin ! I’ve been working in IT for over a decade, mainly as a consultant in the system area ! Head of Opera&ons @ exoscale – Responsible for the overall exoscale plaIorm & product opera&on -
Our Cloud recipe ! To provide efficient public
cloud services, we need : ! A cloud – Cloudstack backed of course ! ! A team – Devops of course ! ! Third party tools ecosystem – Open Source of course ! -
Our Cloudstack based cloud ! Running version 4.0.x
! Linux KVM hypervisor (Ubuntu based) ! Local storage only, no clustering – Keep it Simple, scalable and fast ! Basic networking mode – Means one public IP per VM, No VLANs – Secured by security groups, AWS style -
Opera&ng means also people ! Team must
be kept small – Team growth ra&o != number of virtual machines hosted ! Devops doctrine applied ! Development is opera&ons aware – What impact my code has on produc&on ? ! Opera&ons is managed by development principles and tools – Revert (git) – Documented in line and in commits (avoid rewrite and informa&on loss) – Accountable: who did what -
User Management console Visibility stack Configura7on management system
User backend stack Our Cloudstack Ecoystem -
Exoscale management console ! Homemade management console
! Na&ve cloudstack UI not suitable for our needs ! Simple, fast & efficient ! Hook the cloudstack API ! “AWS style console” -
“User backend stack” ! Homemade user backend modules
! Provide user database, billing/chargeback, &cke&ng and knowledge base ! Used by Cloudstack for user authen&ca&on ! Cloudstack does not offer any na&ve billing/chargeback capability ! Cloudstack only provide usage data which must be processed for billing -
Configura&on management, why ? ! FALSE :
“It is only infrastructure, it does not change” ! Repe&&ve tasks are boring and cost &me ! Small devops team ! Adding & managing more and more – Quickly if required ! ! Deploy, maintain & enforce the same configura&on everywhere ! Adjust con&nuously -
Therefore we need “good ci&zens” ! A machine
should: Automa&cally deploy itself (Almost) Find its iden&ty seings (name, networks,...) Install the necessary packages for which it was intended Register itself to all tools Live along its peers and respect regula&ons Report to cityhall if anything goes wrong -
Configura&on management : Puppet ! We use the
well known open source configura&on management tool Puppet ! Exoscalepuppet : – 40+ modules – Each applica&on got it’s module – Between 10 to 100 commits per week -
Monitoring vs Visibility ! Monitoring is part of
visibility – Tradi&onally: service up, CPU, RAM, network & disk I/O ! Are we genera&ng business value ? – Need more insight into applica&on behavior (who using what, ...) -
Trends ! If it moves, graph it
! If it doesn't move, graph it in case it starts moving ! If it breaks once, monitor it ! Ques&on, adapt and modify thresholds con&nuously -
What is different in the cloud ? !
Distributed systems ! Lots of moving parts ! Scale ! Easy tools to quickly assess produc&on status required -
Visibility stack: Logstash ! Open Source Log collector
! Collects every logs on the machine – Rsyslogd, Cloudstack, Nginx, Load balancer, tomcat, java etc… ! Configura&on managed by puppet ! Backed with Elas&c search cluster -
Visibility stack: Elas&c search / Kibana ! Open
Source distributed resIul search and analy&cs system ! Distributed, NoSQL ! Data indexing is done thru HTTP PUT method, search by HTTP GET ! We store all our logs in a central ES cluster ! Logs kept only 24 hours locally on the server ! Open Source Kibana used as search portal -
Visibility stack: Graphite ! Open Source real-‐&me graphing
system ! Store numeric &me-‐series data ! Render graphs of this data on demand ! Web Dashboard available -
Visibility stack: Riemann ! Open Source monitoring system
! “passive” monitoring, Events stream processor ! Good for monitoring distributed systems ! Well-‐suited for infrastructure with lot of moving parts -
Visibility stack: Collectd ! Open Source Sta&s&cs collector
with many exis&ng plugins ! Plugin may be a script ! Collectd @ exoscale – Metrics sent to graphite – Metrics sent to Riemann – Metrics sent to custom dashboard apps – SNMP polling -
Visibility stack: Command and control ! Private IRC
server & bots used as a « control tower » ! Central view of our infrastrcture : ! Monitoring alarms ! All git commits ! Ability to pilot our servers thru IRC bots: – Puppet apply, apt-‐get, service restart etc… – No need to log on server – Changes can be performed on a group of servers very quickly -
CI: Jenkins ! Jenkins is an Open Source
Con&nuous integra&on server ! Almost every of our apps are built with Jenkins ! Applica&on build may be piloted from IRC ! Linked with IRC and Git ! We build cloudstack with jenkins -
Visibility stack: Dashboards ! Extensive use of dashboards
on TVs ! Graphs, network maps ! Monitoring alerts ! Custom Dashboard applica&on with Cloudstack metrics – Custom Collectd python scripts -
Prac&cal use case ! Add a node to
our Web portal frontend infrastructure : ! Allocate IP/name ! Define machine belonging to the service plaIorm (by a fact) ! “press deploy” ! Let puppet deploy the configura&on and applica&on on the host : – Nginx – Web app ! Let puppet reconfigure load balancing to add this node in the farm ! Watch logs, graph and traffic to this new host in real &me on dashboards -
Conclusion ! “just” installing Cloudstack wasn’t enough for
us ! We’ve built a complete ecosystem around Cloudstack ! Massive automa&on is the key ! Required to be scalable and being operated by small a team ! Give it a try : hsps://portal.exoscale.ch -
References and More ! Links: – hsp://graphite.wikidot.com
– hsp://logstash.net – hsp://Riemann.io – hsp://collectd.org – hsp://puppetlabs.com – hsp://kibana.org/ – hsp://github.com/exoscale/ – hsp://cloudstackcollab.org/