Containerizing your monolith

Transcript

1. Containerizing your monolith Jano González (@janogonzalez)

2. Introduction

3. What’s SoundCloud

4. > 200M Tracks > 20M Creators Many SoundCloud Rappers What’s

   SoundCloud

5. 2017-2018 Migration Keep moving Motivation

6. What’s the balance? Motivation

7. Our migration to microservices The monolith Containerizing the monolith Conclusion

   Contents

8. Our migration to microservices

9. $ rails new soundcloud How we started

10. Around 2012

11. Around 2012

12. Around 2012

13. Around 2012

14. After

15. After

16. But what about deployment?

17. Component Environment Zone Deployment Our abstractions

18. Deployment The process

19. The monolith

20. Track User Playlist The monolith Core entities

21. 360 Chef provisioned bare-metal machines Rails 2.3 Capistrano deployment The

    monolith The technology

22. The monolith The architecture Public API Internal API Public API

    Strangler Internal API caching/ strangler Another BFF Many microservices Public Web Workers

23. The monolith The components Public API MoshiMoshi (Internal API) Public

    Web Assets MoshiMoshi Comments (Internal API) Workers Cron Shell Migration

24. The monolith The hosts component statsd-exporter mtail statsd passenger- exporter

    …

25. Utilization Deployment Lack of confidence The monolith The issues

26. Containerizing the monolith

27. Throw it into Kubernetes

28. Congrats, your monolith is a microservice now

29. Thank You!

30. 1.5 Engineers 1 year until the last bit was cleaned

    up The project

31. The first milestone

32. Docker development container Tests on GoCD The first milestone

33. The proof of concept

34. First staging component The proof of concept Does it even

    work?

35. The proof of concept PR O BLEM ! Init script

    Nginx Passenger Passenger Process Rails App ??? Where are my env variables?

36. The proof of concept Env variables with The Perl Hack™

    SO LU TIO N

37. Productionizing

38. Deployment Monitoring Logs Productionizing Does it run where it matters?

39. Productionizing Anatomy of a traffic serving pod component statsd statsd-exporter

    passenger- exporter mtail twemproxy twemproxy-cu twemproxy- exporter init

40. Don’t choke service discovery Be allocatable Productionizing Sizing the pods

41. CPU units for main container Passenger processes Productionizing Sizing the

    pods 3 16

42. Productionizing PR O BLEM ! Stdout v/s the log metrics

    exporter Component STDOUT Log aggregator File??? Mtail

43. Productionizing Mtail with The Rotatelogs Hack™ SO LU TIO N

44. Productionizing Public API

45. Orchestration Productionizing Public API

46. Productionizing Public API PR O BLEM ! DNS latency and

    our excessive usage Component Service Service Service

47. Productionizing Public API CoreDNS and the DNS Hack™ SO LU

    TIO N

48. Productionizing Internal API

49. Highest throughput Productionizing Internal API

50. Productionizing Internal API PR O BLEM ! Latency was too

    high

51. Productionizing Internal API Optimize GC and make cheaper SQL queries

    SO LU TIO N

52. Productionizing Internal API PR O BLEM ! Errors spikes during

    deployment

53. Productionizing Internal API The preStop Trick™ SO LU TIO N

54. Productionizing Internal API PR O BLEM ! Errors spikes during

    deployment (still???)

55. Productionizing Internal API The Pre Start Trick™ SO LU TIO

    N

56. The rest

57. • Workers • Cron jobs • Shell / Migration hosts

    • Cleanup! The rest

58. Finishing

59. Current status

60. On-prem Cloud Current status Number of pods ~1000 ~140

61. On-prem RPS Cloud RPS Current status Traffic 25K 3K

62. Current status Many deploys

63. Conclusions

64. One Infrastructure One Delivery Process Conclusions What we solved

65. Step by step Controlled rollouts Managing expectations Conclusions How we

    did it

66. Improved utilization Increased confidence Enabling new initiatives Conclusions Benefits

67. Assess current progress Evaluate costs and benefits Conclusions Should you

    do it?

68. Thank You!

69. @janogonzalez https://soundcloud.com/janogonzalez