DevOpsDays Minneapolis 2016: Autoscaling Containers... with Math

DevOpsDays Minneapolis 2016: Autoscaling Containers... with Math

Presented at DevOpsDays Minneapolis. http://www.devopsdays.org/events/2016-minneapolis/program/allan-espinosa/

Also shows how to write your own Autoscaler in Kubernetes

Docker and Kubernetes provide delightful APIs to show various statuses of our applications. Whether CPU, Load average, HTTP response times, etc., we have all that we need to make sure our app is running healthily. When things are on fire, we Ops people twiddle some knobs like spin up more Pods to keep things going. We mostly use our experience and knowledge of the systems that we are running to know what to do.

However, if you look at everyday things like your air conditioner and thermostat, they don't have an Ops team that gets paged to set the correct level of the coolant to set your room to the right temperature. They use some math called Control Theory to keep your room's temperature stable. In this talk, I will show how we can use the same concepts to autoscale and manage the health of our applications on Kubernetes.

93ac6f98100c830506d8d451245635e1?s=128

Allan Espinosa

July 21, 2016
Tweet

Transcript

  1. Autoscaling Containers…with Math @AllanEspinosa

  2. @AllanEspinosa

  3. @AllanEspinosa Your Distributed System Goal OPS Person Twiddle Server Farm

    CPU Utilization Monitoring Page Traffic
  4. @AllanEspinosa Your Aircon Set Temperature Thermostat Valve Coolant Actual Temperature

    Sensor The Weather
  5. @AllanEspinosa Autoscaling Target HPA Number of Pods RC Utilization Heapster

    Utilization Traffic http://kubernetes.io/docs/user-guide/horizontal-pod-autoscaling/
  6. @AllanEspinosa Control Theory • influencing dynamical systems • corrections based

    on feedback loops • math describes effectiveness
  7. @AllanEspinosa R. Routledge, Discoveries & Inventions of the Nineteenth Century,

    13th edition, 1901.
  8. @AllanEspinosa

  9. @AllanEspinosa Control Theory • Target Output • Input • Output

    • Disturbance () + () () () () () − ()
  10. @AllanEspinosa

  11. @AllanEspinosa

  12. @AllanEspinosa Linear-Time Invariant Systems ∂ ∂ () + () =

    () ( + 1) = () + ()
  13. @AllanEspinosa Desired Properties • Stability • Accuracy • Settling time

    • Overshoot
  14. @AllanEspinosa Stability

  15. @AllanEspinosa

  16. @AllanEspinosa

  17. @AllanEspinosa

  18. @AllanEspinosa

  19. @AllanEspinosa Accuracy

  20. @AllanEspinosa Settling time

  21. @AllanEspinosa Overshoot

  22. @AllanEspinosa Controllers () + () () () () () −

    () = () − () () = ∞ ∑ u=−∞ ( − )()
  23. @AllanEspinosa Proportional Control u () = u () () u

    ()
  24. @AllanEspinosa Proportional Control • inherently inaccurate • u increases overshoot

    and settling time
  25. @AllanEspinosa Integral Control u () = u ( − 1)

    + u () • reduce steady-state error • increase settling times
  26. @AllanEspinosa Derivative Control u () = u [() − (

    − 1)] • decrease settling times • sensitive to noise
  27. @AllanEspinosa Non-linearity 40 50 60 70 80 90 100 0.5

    0.7 0.9 1.1 Number of Replicas CPU Utilization
  28. @AllanEspinosa Summary • iterate on feedback • effectiveness of feedback

    • linear models go a long way • re-evaluate your models!
  29. @AllanEspinosa P. Janert, Feedback Control for Computer Systems, Sebastapol, CA:

    O’Reilly Media, 2014. J. Hellerstein, et. al., Feedback Control of Computing Systems, Hoboken, NJ: John Wiley & Sons, 2004.
  30. @AllanEspinosa Thank You! @AllanEspinosa https://github.com/aespinosa/control-theory