Containerizing your monolith

My presentation @ LISA19

Transcript

1. Containerizing your monolith
   Jano González (@janogonzalez)
   

   View Slide

2. Introduction
   

   View Slide

3. What’s SoundCloud
   

   View Slide

4. > 200M Tracks > 20M Creators Many SoundCloud
   Rappers
   What’s SoundCloud
   

   View Slide

5. 2017-2018 Migration Keep moving
   Motivation
   

   View Slide

6. What’s the balance?
   Motivation
   

   View Slide

7. Our migration to microservices
   The monolith
   Containerizing the monolith
   Conclusion
   Contents
   

   View Slide

8. Our migration to microservices
   

   View Slide

9. $ rails new soundcloud
   How we started
   

   View Slide

10. Around 2012
    

    View Slide

11. Around 2012
    

    View Slide

12. Around 2012
    

    View Slide

13. Around 2012
    

    View Slide

14. After
    

    View Slide

15. After
    

    View Slide

16. But what about deployment?
    

    View Slide

17. Component Environment Zone
    Deployment
    Our abstractions
    

    View Slide

18. Deployment
    The process
    

    View Slide

19. The monolith
    

    View Slide

20. Track User Playlist
    The monolith
    Core entities
    

    View Slide

21. 360 Chef provisioned
    bare-metal machines
    Rails 2.3 Capistrano deployment
    The monolith
    The technology
    

    View Slide

22. The monolith
    The architecture
    Public API
    Internal API
    Public API
    Strangler
    Internal API
    caching/
    strangler
    Another BFF
    Many
    microservices
    Public Web
    Workers
    

    View Slide

23. The monolith
    The components
    Public API
    MoshiMoshi
    (Internal API)
    Public Web Assets
    MoshiMoshi
    Comments
    (Internal API)
    Workers Cron Shell Migration
    

    View Slide

24. The monolith
    The hosts
    component
    statsd-exporter
    mtail statsd
    passenger-
    exporter
    …
    

    View Slide

25. Utilization Deployment Lack of confidence
    The monolith
    The issues
    

    View Slide

26. Containerizing the monolith
    

    View Slide

27. Throw it into Kubernetes
    

    View Slide

28. Congrats, your monolith is a
    microservice now
    

    View Slide

29. Thank You!
    

    View Slide

30. 1.5 Engineers 1 year until the last bit
    was cleaned up
    The project
    

    View Slide

31. The first milestone
    

    View Slide

32. Docker development
    container
    Tests on GoCD
    The first milestone
    

    View Slide

33. The proof of concept
    

    View Slide

34. First staging component
    The proof of concept
    Does it even work?
    

    View Slide

35. The proof of concept
    PR
    O
    BLEM
    !
    Init script Nginx Passenger
    Passenger
    Process
    Rails App
    ???
    Where are my env variables?
    

    View Slide

36. The proof of concept
    Env variables with The Perl Hack™
    SO
    LU
    TIO
    N
    

    View Slide

37. Productionizing
    

    View Slide

38. Deployment Monitoring Logs
    Productionizing
    Does it run where it matters?
    

    View Slide

39. Productionizing
    Anatomy of a traffic serving pod
    component statsd statsd-exporter
    passenger-
    exporter
    mtail twemproxy
    twemproxy-cu
    twemproxy-
    exporter
    init
    

    View Slide

40. Don’t choke service
    discovery
    Be allocatable
    Productionizing
    Sizing the pods
    

    View Slide

41. CPU units for main
    container
    Passenger processes
    Productionizing
    Sizing the pods
    3 16
    

    View Slide

42. Productionizing
    PR
    O
    BLEM
    !
    Stdout v/s the log metrics exporter
    Component
    STDOUT
    Log aggregator
    File???
    Mtail
    

    View Slide

43. Productionizing
    Mtail with The Rotatelogs Hack™
    SO
    LU
    TIO
    N
    

    View Slide

44. Productionizing Public API
    

    View Slide

45. Orchestration
    Productionizing Public API
    

    View Slide

46. Productionizing Public API
    PR
    O
    BLEM
    !
    DNS latency and our excessive usage
    Component
    Service Service Service
    

    View Slide

47. Productionizing Public API
    CoreDNS and the DNS Hack™
    SO
    LU
    TIO
    N
    

    View Slide

48. Productionizing Internal API
    

    View Slide

49. Highest throughput
    Productionizing Internal API
    

    View Slide

50. Productionizing Internal API
    PR
    O
    BLEM
    !
    Latency was too high
    

    View Slide

51. Productionizing Internal API
    Optimize GC and make cheaper SQL queries
    SO
    LU
    TIO
    N
    

    View Slide

52. Productionizing Internal API
    PR
    O
    BLEM
    !
    Errors spikes during deployment
    

    View Slide

53. Productionizing Internal API
    The preStop Trick™
    SO
    LU
    TIO
    N
    

    View Slide

54. Productionizing Internal API
    PR
    O
    BLEM
    !
    Errors spikes during deployment (still???)
    

    View Slide

55. Productionizing Internal API
    The Pre Start Trick™
    SO
    LU
    TIO
    N
    

    View Slide

56. The rest
    

    View Slide

57. ● Workers
    
    ● Cron jobs
    
    ● Shell / Migration hosts
    
    ● Cleanup!
    The rest
    

    View Slide

58. Finishing
    

    View Slide

59. Current status
    

    View Slide

60. On-prem Cloud
    Current status
    Number of pods
    ~1000 ~140
    

    View Slide

61. On-prem RPS Cloud RPS
    Current status
    Traffic
    25K 3K
    

    View Slide

62. Current status
    Many deploys
    

    View Slide

63. Conclusions
    

    View Slide

64. One Infrastructure One Delivery Process
    Conclusions
    What we solved
    

    View Slide

65. Step by step Controlled rollouts Managing expectations
    Conclusions
    How we did it
    

    View Slide

66. Improved utilization Increased confidence Enabling new initiatives
    Conclusions
    Benefits
    

    View Slide

67. Assess current progress Evaluate costs and
    benefits
    Conclusions
    Should you do it?
    

    View Slide

68. Thank You!
    

    View Slide

69. @janogonzalez
    https://soundcloud.com/janogonzalez
    

    View Slide