AWS SQS queues & Kubernetes Autoscaling Pitfalls Stories

3528f569ab96088a8a2b00b3c33de599?s=47 Eric Khun
October 26, 2020

AWS SQS queues & Kubernetes Autoscaling Pitfalls Stories

Talk at the Cloud Native Computing Foundation meetup @dcard.tw

3528f569ab96088a8a2b00b3c33de599?s=128

Eric Khun

October 26, 2020
Tweet

Transcript

  1. AWS SQS queues & Kubernetes Autoscaling Pitfalls Stories Cloud Native

    Foundation meetup @dcard.tw @eric_khun
  2. Make it work, Make it right, Make it fast kent

    beck (agile manifesto - extreme programming)
  3. Make it work, Make it right, Make it fast kent

    beck (agile manifesto - extreme programming)
  4. Make it work, Make it right, Make it fast kent

    beck (agile manifesto - extreme programming)
  5. Buffer

  6. None
  7. Buffer • 80 employees , 12 time zones, all remote

  8. Quick intro

  9. None
  10. Main pipelines flow

  11. it can look like ... golang Talk @Maicoin :

  12. None
  13. None
  14. How do we send posts to social medias?

  15. A bit of history... 2010 -> 2012: Joel (founder/ceo) 1

    cronjob on a Linode server $20/mo 512 mb of RAM 2012 -> 2017 : Sunil (ex-CTO) Crons running on AWS ElasticBeanstalk / supervisord 2017 -> now: Kubernetes / CronJob controller
  16. AWS Elastic Beanstalk: Kubernetes:

  17. At what scale? ~ 3 million SQS messages per hour

  18. Different patterns for many queues

  19. Are our workers (consumers of the SQS queues ) efficients?

  20. Are our workers efficients?

  21. Are our workers efficients?

  22. Empty messages? > Workers tries to pull messages from SQS,

    but receive “nothing” to process
  23. Number of empty messages per queue

  24. Sum of empty messages on all queues

  25. None
  26. 1,000,000 API calls to AWS costs 0.40$ We have 7,2B

    calls/month for “empty messages” It costs ~$25k/year > Me:
  27. None
  28. AWS SQS Doc

  29. None
  30. Or in the AWS console

  31. Results?

  32. empty messages

  33. AWS

  34. None
  35. $120 > $50 saved daily > $2000 / month >

    $25,000 / year (it’s USD, not TWD)
  36. Paid for querying “nothing”

  37. (for the past 8 years )

  38. Benefits - Saving money - Less CPU usage (less empty

    requests) - Less throttling (misleading) - Less containers > Better resources allocation: memory/cpu request
  39. Why did that happen?

  40. Default options

  41. None
  42. Never questioning what’s working decently or the way it’s been

    always done
  43. What could have helped? Infra as code (explicit options /

    standardization) SLI/SLOs (keep re-evaluating what’s important) AWS architecture reviews (taging/recommendations from aws solutions architects)
  44. Make it work, Make it right, Make it fast

  45. Make it work, Make it right, Make it fast

  46. Do you remember?

  47. None
  48. None
  49. None
  50. Need to analytics on Twitter/FB/IG/LKD… on millions on posts faster

  51. workers consuming time

  52. None
  53. What’s the problem?

  54. Resources allocated and not doing anything most of the time

    Developer trying to put find compromises on the number of workers
  55. How to solve it?

  56. Autoscaling! (with Keda.sh) Supported by IBM / Redhat / Microsoft

  57. None
  58. Results

  59. None
  60. But notice anything?

  61. Before autoscaling

  62. After autoscaling

  63. After autoscaling

  64. What’s happening?

  65. Downscaling

  66. Why?

  67. delete pod lifecycle

  68. what went wrong - Workers didn’t manage SIGTERM sent by

    k8s - Kept processing messages - Messages were halfway processed and killed - Messages were sent back to the the queue again - Less workers because of downscaling
  69. solution - When receiving SIGTERM stop processing new messages -

    Set a graceful period long enough to process the current message if (SIGTERM) { // finish current processing and stop receiving new messages }
  70. None
  71. None
  72. And it can also help with sqs empty messages

  73. Make it work, Make it right, Make it fast

  74. Make it work, Make it right, Make it fast

  75. Thanks!

  76. Questions? monitory.io taiwangoldcard.com travelhustlers.co ✈