AWS SQS queues & Kubernetes Autoscaling Pitfalls Stories

Transcript

1. AWS SQS queues & Kubernetes Autoscaling Pitfalls Stories Cloud Native

   Foundation meetup @dcard.tw @eric_khun

2. Make it work, Make it right, Make it fast kent

   beck (agile manifesto - extreme programming)

3. Make it work, Make it right, Make it fast kent

   beck (agile manifesto - extreme programming)

4. Make it work, Make it right, Make it fast kent

   beck (agile manifesto - extreme programming)

5. Buffer

6. None

7. Buffer • 80 employees , 12 time zones, all remote

8. Quick intro

9. None

10. Main pipelines flow

11. it can look like ... golang Talk @Maicoin :

12. None
13. None

14. How do we send posts to social medias?

15. A bit of history... 2010 -> 2012: Joel (founder/ceo) 1

    cronjob on a Linode server $20/mo 512 mb of RAM 2012 -> 2017 : Sunil (ex-CTO) Crons running on AWS ElasticBeanstalk / supervisord 2017 -> now: Kubernetes / CronJob controller

16. AWS Elastic Beanstalk: Kubernetes:

17. At what scale? ~ 3 million SQS messages per hour

18. Different patterns for many queues

19. Are our workers (consumers of the SQS queues ) efficients?

20. Are our workers efficients?

21. Are our workers efficients?

22. Empty messages? > Workers tries to pull messages from SQS,

    but receive “nothing” to process

23. Number of empty messages per queue

24. Sum of empty messages on all queues

25. None

26. 1,000,000 API calls to AWS costs 0.40$ We have 7,2B

    calls/month for “empty messages” It costs ~$25k/year > Me:
27. None

28. AWS SQS Doc

29. None

30. Or in the AWS console

31. Results?

32. empty messages

33. AWS

34. None

35. $120 > $50 saved daily > $2000 / month >

    $25,000 / year (it’s USD, not TWD)

36. Paid for querying “nothing”

37. (for the past 8 years )

38. Benefits - Saving money - Less CPU usage (less empty

    requests) - Less throttling (misleading) - Less containers > Better resources allocation: memory/cpu request

39. Why did that happen?

40. Default options

41. None

42. Never questioning what’s working decently or the way it’s been

    always done

43. What could have helped? Infra as code (explicit options /

    standardization) SLI/SLOs (keep re-evaluating what’s important) AWS architecture reviews (taging/recommendations from aws solutions architects)

44. Make it work, Make it right, Make it fast

45. Make it work, Make it right, Make it fast

46. Do you remember?

47. None
48. None
49. None

50. Need to analytics on Twitter/FB/IG/LKD… on millions on posts faster

51. workers consuming time

52. None

53. What’s the problem?

54. Resources allocated and not doing anything most of the time

    Developer trying to put find compromises on the number of workers

55. How to solve it?

56. Autoscaling! (with Keda.sh) Supported by IBM / Redhat / Microsoft

57. None

58. Results

59. None

60. But notice anything?

61. Before autoscaling

62. After autoscaling

63. After autoscaling

64. What’s happening?

65. Downscaling

66. Why?

67. delete pod lifecycle

68. what went wrong - Workers didn’t manage SIGTERM sent by

    k8s - Kept processing messages - Messages were halfway processed and killed - Messages were sent back to the the queue again - Less workers because of downscaling

69. solution - When receiving SIGTERM stop processing new messages -

    Set a graceful period long enough to process the current message if (SIGTERM) { // finish current processing and stop receiving new messages }
70. None
71. None

72. And it can also help with sqs empty messages

73. Make it work, Make it right, Make it fast

74. Make it work, Make it right, Make it fast

75. Thanks!

76. Questions? monitory.io taiwangoldcard.com travelhustlers.co ✈