Pro Yearly is on sale from $80 to $50! »

DevOpsDays Silicon Valley 2016: Autoscaling Containers... with Math

DevOpsDays Silicon Valley 2016: Autoscaling Containers... with Math

Presented at DevOpsDays Silicon Valley 2016 http://dodsv2016.busyconf.com/schedule#activity_57282a2fd4bb1947a60000a8

Docker and Kubernetes provide delightful APIs to show various statuses of our applications. Whether CPU, Load average, HTTP response times, etc., we have all that we need to make sure our app is running healthily. When things are on fire, we Ops people twiddle some knobs like spin up more Pods to keep things going. We mostly use our experience and knowledge of the systems that we are running to know what to do.

However, if you look at everyday things like your air conditioner and thermostat, they don't have an Ops team that gets paged to set the correct level of the coolant to set your room to the right temperature. They use some math called Control Theory to keep your room's temperature stable. In this talk, I will show how we can use the same concepts to autoscale and manage the health of our applications on Kubernetes.

93ac6f98100c830506d8d451245635e1?s=128

Allan Espinosa

June 24, 2016
Tweet

Transcript

  1. Autoscaling Containers…with Math @AllanEspinosa

  2. @AllanEspinosa

  3. @AllanEspinosa

  4. @AllanEspinosa Ruby on Rails • Redis • RDS • Node.js

    • Chef • Nginx • PHP Built and proven on over 1 Billion AWS hours. Reliability and support you can trust. We ARE DevOps use promo code “DEVOPSDAYS2016” for 500 free trial hours www.engineyard.com/trial
  5. @AllanEspinosa Your Distributed System Goal OPS Person Twiddle Server Farm

    CPU Utilization Monitoring Page Traffic
  6. @AllanEspinosa Your Aircon Set Temperature Thermostat Valve Coolant Actual Temperature

    Sensor The Weather
  7. @AllanEspinosa Support questions • How many unicorn workers should I

    have? • Why is monit killing my sidekiq workers?
  8. @AllanEspinosa

  9. @AllanEspinosa Autoscaling Target HPA Number of Pods RC Utilization Heapster

    Utilization Traffic http://kubernetes.io/docs/user-guide/horizontal-pod-autoscaling/
  10. @AllanEspinosa Control Theory • influencing dynamical systems • corrections based

    on feedback loops • math describes effectiveness
  11. @AllanEspinosa R. Routledge, Discoveries & Inventions of the Nineteenth Century,

    13th edition, 1901.
  12. @AllanEspinosa The Royal Society is collaborating with JSTOR to digitize,

    preserve, and extend access to Proceedings of the Royal Society of London. www.jsto
  13. @AllanEspinosa Control Theory • Target Output • Input • Output

    • Disturbance () + () () () () () − ()
  14. @AllanEspinosa • Regulatory Control • Disturbance rejection 0 0.1 0.2

    0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 2000 4000 6000 8000 10000 0 100 200 300 400 500 600 CPU Utilization Number of Instances Time (s) CPU Utilization Number of Instances
  15. @AllanEspinosa Linear-Time Invariant Systems ∂ ∂ () + () =

    () ( + 1) = () + ()
  16. @AllanEspinosa Desired Properties • Stability • Accuracy • Settling time

    • Overshoot
  17. @AllanEspinosa Stability

  18. @AllanEspinosa

  19. @AllanEspinosa

  20. @AllanEspinosa

  21. @AllanEspinosa

  22. @AllanEspinosa Accuracy

  23. @AllanEspinosa Settling time

  24. @AllanEspinosa Overshoot

  25. @AllanEspinosa Controllers () + () () () () () −

    () = () − () () = ∞ ∑ u=−∞ ( − )()
  26. @AllanEspinosa Proportional Control u () = u () () u

    ()
  27. @AllanEspinosa Proportional Control • inherently inaccurate • u increases overshoot

    and settling time
  28. @AllanEspinosa Integral Control u () = u ( − 1)

    + u () • reduce steady-state error • increase settling times
  29. @AllanEspinosa Derivative Control u () = u [() − (

    − 1)] • decrease settling times • sensitive to noise
  30. @AllanEspinosa Non-linearity 0 0.2 0.4 0.6 0.8 1 0 20

    40 60 80 100 120 CPU Utilization Number of Instances
  31. @AllanEspinosa Summary • iterate on feedback • effectiveness of feedback

    • linear models go a long way • re-evaluate your models!
  32. @AllanEspinosa

  33. @AllanEspinosa Fill out our survey by 3PM on the final

    day of the conference for a chance to win: Fighter Building Kit ey.io/EYprize
  34. Thank You! @AllanEspinosa https://ey.io/EYprize https://github.com/aespinosa/control-theory