Introducing Vamana j.mp/to-vamana 

About Me ● Dev/Ops @ Indix ● Hindu Mythology Fan ● OSS Contributor ● ashwanthkumar.in 

init Auto scalars are init systems in your data center 

Enter AWS 

● Scales massively ● Has Cool Off capabilities to avoid scale storms ● Auto Rebalancing across AZs / Subnets ● Integration with ELB Amazon ASG - Good Parts 

Amazon ASG - Limitations ● Can support only 1 launch configuration actively ○ Single Instance type ○ Single life cycle - Spot / OD ● Auto Scaling Tightly coupled with only Cloudwatch ● Alarms can be triggered (automatically) but only based on a single metric ● Limited stat functions - avg, sum, min, max and # data samples 

Enter Vamana http://github.com/ashwanthkumar/vamana2 

Vamana Architecture Vamana - Collect Demand and Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value 

Vamana for Hadoop1 Vamana - Collect Demand and Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push MR Demand and Supply Metrics Get MR Demand And Supply Metrics Set the computed “Desired” Value 

Demand vs Supply Metrics for Hadoop ● We collect supply metrics from the Cluster Summary table ○ map_supply ○ reduce_supply ● Demand metrics are collected as cumulative sum of map & reduce tasks of all Running jobs ○ map_demand ○ reduce_demand https://github.com/ashwanthkumar/hadoop-as-publisher 

Demand vs Supply Metrics for Hadoop 

Vamana - Configuration 

After Vamana - Demand vs Supply 

After Vamana - Cost Reduction Savings ~ $ 30 per day 

Vamana Vamana - Collect Demand and Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value 

Vamana - Pluggable App Scalar Vamana - Collect Demand and Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value 

Vamana - Pluggable Metric Store Vamana - Collect Demand and Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value 

Vamana - Pluggable Auto scalar Vamana - Collect Demand and Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value 

Questions? Thank you! https://github.com/ashwanthkumar/vamana2 

Meta Implementation Details and other notes 

Metric Collector 

App Scalar 

Hadoop Scalar 

Auto scalar 

● We’ve lots of datapipelines running on various versions of Hadoop ● Each of them have their own usage pattern ○ A Staging cluster has only workloads for 3-4 hours a day ○ Production cluster has workloads 24x7 ● We started having Scale Up and Scale Down stages in our Datapipelines ● Initially it helped but started breaking when ○ Pipeline fails before completion and the cluster is not scaled down ○ Every new pipeline created had to have a scale up and scale down stage ○ More than 1 pipeline started sharing the cluster Hadoop @ Indix