Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introducing Vamana

Introducing Vamana

As presented on Chennai AWS Meetup - October 2015

Event - http://www.meetup.com/Chennai-Amazon-Web-Services-Meetup/events/226182749/

Ashwanth Kumar

October 31, 2015
Tweet

More Decks by Ashwanth Kumar

Other Decks in Technology

Transcript

  1. About Me • Dev/Ops @ Indix • Hindu Mythology Fan

    • OSS Contributor • ashwanthkumar.in
  2. • Scales massively • Has Cool Off capabilities to avoid

    scale storms • Auto Rebalancing across AZs / Subnets • Integration with ELB Amazon ASG - Good Parts
  3. Amazon ASG - Limitations • Can support only 1 launch

    configuration actively ◦ Single Instance type ◦ Single life cycle - Spot / OD • Auto Scaling Tightly coupled with only Cloudwatch • Alarms can be triggered (automatically) but only based on a single metric • Limited stat functions - avg, sum, min, max and # data samples
  4. Vamana Architecture Vamana - Collect Demand and Supply metrics from

    CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value
  5. Vamana for Hadoop1 Vamana - Collect Demand and Supply metrics

    from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push MR Demand and Supply Metrics Get MR Demand And Supply Metrics Set the computed “Desired” Value
  6. Demand vs Supply Metrics for Hadoop • We collect supply

    metrics from the Cluster Summary table ◦ map_supply ◦ reduce_supply • Demand metrics are collected as cumulative sum of map & reduce tasks of all Running jobs ◦ map_demand ◦ reduce_demand https://github.com/ashwanthkumar/hadoop-as-publisher
  7. Vamana Vamana - Collect Demand and Supply metrics from CloudWatch

    / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value
  8. Vamana - Pluggable App Scalar Vamana - Collect Demand and

    Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value
  9. Vamana - Pluggable Metric Store Vamana - Collect Demand and

    Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value
  10. Vamana - Pluggable Auto scalar Vamana - Collect Demand and

    Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value
  11. • We’ve lots of datapipelines running on various versions of

    Hadoop • Each of them have their own usage pattern ◦ A Staging cluster has only workloads for 3-4 hours a day ◦ Production cluster has workloads 24x7 • We started having Scale Up and Scale Down stages in our Datapipelines • Initially it helped but started breaking when ◦ Pipeline fails before completion and the cluster is not scaled down ◦ Every new pipeline created had to have a scale up and scale down stage ◦ More than 1 pipeline started sharing the cluster Hadoop @ Indix