Orchestrating Clusters with Ironfan and Chef

How you can represent your complete cluster with one config file and have it deployed to Cloud or Bare Metal. Infochmimps’ Ironfan builds on Opscode Chef to allow you to specify and orchestrate all flavors of your cluster’s deployment, monitoring and growth. Not just the core HBase/HDFS/MapReduce/Hive/Flume, etc. but all the elements including web / app servers, mysql, redis, rabbitmq and whatever other servers needed to implement your service. These same tools can manage variations for development, staging, R&D as well as the target “rendering” to various Clouds, Bare Metal or even Vagrant VMs.

Transcript

1. Orchestrating Clusters with Ironfan and Chef Robert J. Berger - CTO Runa, Inc. rberger@runa.com 1 Sunday, March 31, 13
2. Hassles of Big Data Stack Deployments • Lots of Moving Parts • Hadoop/HBase just one sub-system • Heterogeneous Tech • Monitoring & Metrics Everywhere • Details obscure the big picture • Need repeatability & variations on themes 2 Sunday, March 31, 13
3. The Forest for the Trees 3 Sunday, March 31, 13
4. Forest Session Store HBase MySQL Dash board App Servers Monitor Hadoop M/R AMQP Elastic Search 4 Sunday, March 31, 13
5. Trees Web Rails App Rails App ELB Web Web Clj App Clj App Clj App Web Graphite Ganglia Logstash Sensu Statsd GDash ES Server ES Server ES Server ES Server Rabbit Rabbit Redis Redis Redis ZooKeeper ZooKeeper ZooKeeper HB Master HB Master Regionsrvr Regionsrvr Regionsrvr Regionsrvr Regionsrvr Regionsrvr Regionsrvr Regionsrvr Regionsrvr Regionsrvr Regionsrvr Master Sec Master Slave Slave Slave Slave Slave Slave Slave Slave Slave Slave Master Slave Slave Dashboard App Servers Monitoring Elastic Search AMQP Hadoop M/R Session Store HBase MySQL 5 Sunday, March 31, 13
6. Leaves Dashboard App Servers Monitoring Elastic Search AMQP Hadoop M/R Session Store HBase MySQL Nginx Reverse Proxy Unicorn Rails Dashboard App Java Postfix Cron jobs Sensu client Sensu plugins Elastic Load Balancer Nginx Reverse Proxy Swarmiji Java Leiningen Jark Clojure Apps HBase Client Postfix Cron jobs Sensu client Sensu plugins MySQL Client Upstart Config Logstash Client MySQL Client Upstart Config Logstash Client Nginx Reverse Proxy Java Leiningen Jark Postfix Cron jobs Sensu Server Sensu Web Sensu Client Sensu Plugins Ganglia Server Ganglia Web Statsd Server Graphite Server Grpahite Web Python Logstash MySQL Client Upstart Config Logstash Client Elastic Search Server Java Cron jobs Sensu client Sensu plugins Upstart Config RabbitMQ RabbitMQ Plugins Cluster Config Erlang Cron jobs Sensu client Sensu plugins Upstart Config Namenode Secondary Namenode Tasktracker Jobtrackers Bootstrap Namenmode Java JMX Cron jobs Sensu client Sensu plugins Ganglia Client Upstart Config Namenode Secondary Namenode Tasktracker Jobtrackers Datanodes Bootstrap Namenmode Java JMX Cron jobs Sensu client Sensu plugins Ganglia Client Upstart Config Redis Cron jobs Sensu client Sensu plugins Upstart Config Zookeeper HBase Master Regionserver MySQL Master MySQL Slaves Cluster Setup Cron jobs Sensu client Sensu plugins Upstart Config 6 Sunday, March 31, 13
7. Molecules <configuration> <property> <name>hbase.rootdir</name> <value>hdfs://ip-10-17-57-58.ec2.internal:8020/hadoop/hbase</value> <description>The directory shared by region servers. Should be fully-qualified to include the filesystem to use. E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR </description> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> <description>The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh) </description> </property> <property> <name>hbase.zookeeper.quorum</name> <value>master0-cluster0.runa.com,regionserver0- cluster0.runa.com,regionserver1-cluster0.runa.com</value> <description>Comma separated list of servers in the ZooKeeper Quorum. For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com". By default this is set to localhost for local and pseudo-distributed modes of operation. For a fully-distributed setup, this should be set to a full list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh this is the list of servers which we will start/stop ZooKeeper on. </description> </property> 7 Sunday, March 31, 13
8. Config Management: Leaves & Molecules • Chef, Puppet, Cfengine • Much better than shell scripts or cli jocks • Infrastructure as code • Still No Forest Perspective 8 Sunday, March 31, 13
9. Ironfan: Forest, Trees, Leaves and Molecules • Builds on top of Chef • Cluster Description in Single File • Your System Diagram Come to Life • Components announce capabilities • Service Discovery automates interconnects • Knife (CLI) extension controls cluster and component life-cycles 9 Sunday, March 31, 13
10. Community + Own Cookbooks VMs and/or Servers Running Chef-Client Chef Development Host N N N N Basic Chef Chef Knife: Upload Cookboks to Chef Server Launch Instances / Bootstrap Servers Chef Server Roles / Cookbooks Data Bags Nodes Attributes Auth & ACLs Search 10 Sunday, March 31, 13
11. Nodes Attributes Search Chef + Ironfan Community + Own Cookbooks VMs and/or Servers Running Chef-Client Chef Development Host Chef Knife + Ironfan Gem: Launch / Bootstrap / Manage Whole Clusters, All Facets or Specific Instances N N N N Chef Server Roles / Cookbooks Data Bags Auth & ACLs 11 Sunday, March 31, 13
12. Nodes Attributes Search Chef + Ironfan Community + Own Cookbooks VMs and/or Servers Running Chef-Client Chef Development Host Ironfan Pantry + Chef Knife + Ironfan Gem: Launch / Bootstrap / Manage Whole Clusters, All Facets or Specific Instances N N N N Chef Server Roles / Cookbooks Data Bags Nodes Attributes Discovery Auth & ACLs Search / Discovery 11 Sunday, March 31, 13
13. Ironfan Components • Ironfan Gem: • Knife Plugins to orchestrate clusters • Logic to sync Chef Server & Cloud[s] • Silverware: Coordinate Discovery of Services • Ironfan-Homebase: Ironfan tuned Chef-Repo • Ironfan-Pantry: Cookbooks tuned for Clusters • Ironfan-CI: Testing of Ironfan clusters and Cookbooks 12 Sunday, March 31, 13
14. Cluster Config: Forest View facet 'master' do instances 1 cloud.image_id 'ami-93c31afa' cloud.flavor "cc1.4xlarge" role "big_package" role "hadoop_master" role "hbase_master" recipe "cluster_chef::cluster_webfront" recipe "hbase::utils" recipe "route53::runa" role "monitored_client" end Global Cloud & Cluster Configs Facets are the “Trees” Roles & Recipes are the “Leaves” ClusterChef.cluster 'base0-cluster0' do setup_role_implications cloud :ec2 do region 'us-east-1' availability_zones ['us-east-1b'] backing 'ebs' image_name 'natty' security_group(cluster_root) do authorize_port_range(22) end end role "base_role" role "chef_client" role "base0-cluster0" role "production" role "runastack" role "ebs_volumes_attach" role "ebs_volumes_mount" 13 Sunday, March 31, 13
15. Global Cloud & Recipe Confgs ClusterChef.cluster 'base0-cluster0' do setup_role_implications cloud :ec2 do region 'us-east-1' availability_zones ['us-east-1b'] backing 'ebs' image_name 'natty' security_group(cluster_root) do authorize_port_range(22) end end role "base_role" role "chef_client" role "base0-cluster0" role "production" role "runastack" role "ebs_volumes_attach" role "ebs_volumes_mount" Cluster Name Cloud Configs Shared Roles Configure Security Group 14 Sunday, March 31, 13
16. Facets: Server Objects • Describes top level functionality • Roles, Recipes & main Attribute overrides • How many to instantiate • Target deployment specifics • Cloud specifics like AMI, Region, etc. 15 Sunday, March 31, 13
17. Facets add Specifics facet 'master' do instances 1 cloud.image_id 'ami-93c31afa' cloud.flavor "cc1.4xlarge" role "hadoop_master" role "hbase_master" recipe "cluster_chef::cluster_webfront" recipe "hbase::utils" recipe "route53::runa" role "monitored_client" end facet 'regionserver' do instances 7 cloud.image_id 'ami-93c31afa' cloud.flavor "cc1.4xlarge" role "hadoop_slave" role "hbase_regionserver" recipe "hbase::utils" recipe "route53::runa" role "monitored_client" server 0 do role "zookeeper_server" end end Facet Name Facet Name Number of Copies Number of Copies Cloud Overrides Facet Roles & Recipes Facet Roles & Recipes Cloud Overrides Make one instance special 16 Sunday, March 31, 13
18. Facets Composed of Components • Components are Services • Nginx, MySQL server, Zookeeper, HBMaster, Namenode, etc. • Chef Cookbooks manage components • Ironfan Silverware for service discovery • Auto-Connects components together 17 Sunday, March 31, 13
19. Silverware: Service Discovery • Announcements enable Service Discovery • Implemented using Chef Node Attributes • DSL for Chef Attributes • Components announce Services & Amenities • Enables zero-conf for things like Logging, Ports, Dashboard sources, etc. • Announcements effectively define a component's Contract or API 18 Sunday, March 31, 13
20. Silverware Service Discovery hbase_config = Mash.new({ :namenode_fqdn => discover(:hadoop, :namenode ).private_hostname), :jobtracker_addr => discover(:hadoop, :jobtracker).private_ip), :zookeeper_addrs => discover_all(:zookeeper, :server).map(&:private_ip).sort, :ganglia => discover(:ganglia, :server), :ganglia_addr => discover(:ganglia, :server).private_hostname), :private_ip => private_ip_of(node) }) announce(:hadoop, :namenode) • Recipe that creates a service , announces it • Recipe that requires a service , discovers it 19 Sunday, March 31, 13
21. Silverware: Announce Capabilities • Goes beyond basic component discovery • Capabilities • Serve webpages, execute script, send metrics, answer queries • Aspects • Ports, daemons, logs, files, dashboards 20 Sunday, March 31, 13
22. Aspects enable Zeroconf Amenities • A log aspect would enable the following amenities • logrotated to manage its logs • flume to archive logs to a location • A port aspect would enable • Configuration of firewall • Monitoring of port uptime & latency • Remote checks that firewalled ports do NOT respond 21 Sunday, March 31, 13
23. Knife Cluster: Lifecycle Management • A Plugin for Opscode Chef Knife • Deployment & Lifecycle Operations: • launch, bootstrap, kill, start, stop • Access and Chef Operations • ssh, kick, proxy • Utilities • show, sync 22 Sunday, March 31, 13
24. Knife Launch Cluster, Facet or Instance[s] • Launch all the nodes in a cluster knife cluster launch base0-master0 • Launch just a single instance of a single facet knife cluster launch base0-master0 master 0 • Launch all the instances of a facet knife cluster launch base0-master0 regionserver 23 Sunday, March 31, 13
25. Stop/Start Cluster, Facet or Instance[s] • Stop whole cluster knife cluster stop base0-master0 • Stop a single instance of a single facet knife cluster stop base0-master0 master 0 • Stop all instances of a facet knife cluster stop base0-master0 regionserver 24 Sunday, March 31, 13
26. Same Knife Command to Launch Vagrant[s] • Can use the same cluster configurations and knife command to launch Vagrants knife cluster vagrant up base0-master0 • Still Experimental 25 Sunday, March 31, 13
27. Ironfan-CI • Jenkins based Continuous Integration of Clusters • Still Experimental • Uses Discovery to automate baseline test creation • Leverages Vagrant to create clean test environments 26 Sunday, March 31, 13
28. References • Basic Chef Stuff: http://wiki.opscode.com/display/chef/Home • Ironfan Screencast: http://vimeo.com/37279372 • Ironfan Wiki for the most complete info : https://github.com/infochimps-labs/ironfan/wiki • The Forest for the Trees Photo - Ame Otoko http://www.flickr.com/photos/ameotoko/5383225925/ 27 Sunday, March 31, 13