Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Containerizing your monolith

Containerizing your monolith

My presentation @ LISA19

Jano González

October 30, 2019

More Decks by Jano González

Other Decks in Programming


  1. Containerizing your monolith Jano González (@janogonzalez)

  2. Introduction

  3. What’s SoundCloud

  4. > 200M Tracks > 20M Creators Many SoundCloud Rappers What’s

  5. 2017-2018 Migration Keep moving Motivation

  6. What’s the balance? Motivation

  7. Our migration to microservices The monolith Containerizing the monolith Conclusion

  8. Our migration to microservices

  9. $ rails new soundcloud How we started

  10. Around 2012

  11. Around 2012

  12. Around 2012

  13. Around 2012

  14. After

  15. After

  16. But what about deployment?

  17. Component Environment Zone Deployment Our abstractions

  18. Deployment The process

  19. The monolith

  20. Track User Playlist The monolith Core entities

  21. 360 Chef provisioned bare-metal machines Rails 2.3 Capistrano deployment The

    monolith The technology
  22. The monolith The architecture Public API Internal API Public API

    Strangler Internal API caching/ strangler Another BFF Many microservices Public Web Workers
  23. The monolith The components Public API MoshiMoshi (Internal API) Public

    Web Assets MoshiMoshi Comments (Internal API) Workers Cron Shell Migration
  24. The monolith The hosts component statsd-exporter mtail statsd passenger- exporter

  25. Utilization Deployment Lack of confidence The monolith The issues

  26. Containerizing the monolith

  27. Throw it into Kubernetes

  28. Congrats, your monolith is a microservice now

  29. Thank You!

  30. 1.5 Engineers 1 year until the last bit was cleaned

    up The project
  31. The first milestone

  32. Docker development container Tests on GoCD The first milestone

  33. The proof of concept

  34. First staging component The proof of concept Does it even

  35. The proof of concept PR O BLEM ! Init script

    Nginx Passenger Passenger Process Rails App ??? Where are my env variables?
  36. The proof of concept Env variables with The Perl Hack™

  37. Productionizing

  38. Deployment Monitoring Logs Productionizing Does it run where it matters?

  39. Productionizing Anatomy of a traffic serving pod component statsd statsd-exporter

    passenger- exporter mtail twemproxy twemproxy-cu twemproxy- exporter init
  40. Don’t choke service discovery Be allocatable Productionizing Sizing the pods

  41. CPU units for main container Passenger processes Productionizing Sizing the

    pods 3 16
  42. Productionizing PR O BLEM ! Stdout v/s the log metrics

    exporter Component STDOUT Log aggregator File??? Mtail
  43. Productionizing Mtail with The Rotatelogs Hack™ SO LU TIO N

  44. Productionizing Public API

  45. Orchestration Productionizing Public API

  46. Productionizing Public API PR O BLEM ! DNS latency and

    our excessive usage Component Service Service Service
  47. Productionizing Public API CoreDNS and the DNS Hack™ SO LU

    TIO N
  48. Productionizing Internal API

  49. Highest throughput Productionizing Internal API

  50. Productionizing Internal API PR O BLEM ! Latency was too

  51. Productionizing Internal API Optimize GC and make cheaper SQL queries

  52. Productionizing Internal API PR O BLEM ! Errors spikes during

  53. Productionizing Internal API The preStop Trick™ SO LU TIO N

  54. Productionizing Internal API PR O BLEM ! Errors spikes during

    deployment (still???)
  55. Productionizing Internal API The Pre Start Trick™ SO LU TIO

  56. The rest

  57. • Workers • Cron jobs • Shell / Migration hosts

    • Cleanup! The rest
  58. Finishing

  59. Current status

  60. On-prem Cloud Current status Number of pods ~1000 ~140

  61. On-prem RPS Cloud RPS Current status Traffic 25K 3K

  62. Current status Many deploys

  63. Conclusions

  64. One Infrastructure One Delivery Process Conclusions What we solved

  65. Step by step Controlled rollouts Managing expectations Conclusions How we

    did it
  66. Improved utilization Increased confidence Enabling new initiatives Conclusions Benefits

  67. Assess current progress Evaluate costs and benefits Conclusions Should you

    do it?
  68. Thank You!

  69. @janogonzalez https://soundcloud.com/janogonzalez