Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Containerizing your monolith
Search
Jano González
October 30, 2019
Programming
530
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Containerizing your monolith
My presentation @ LISA19
Jano González
October 30, 2019
Transcript
69 slides
Containerizing your monolith Jano González (@janogonzalez)
Introduction
What’s SoundCloud
> 200M Tracks > 20M Creators Many SoundCloud Rappers What’s
SoundCloud
2017-2018 Migration Keep moving Motivation
What’s the balance? Motivation
Our migration to microservices The monolith Containerizing the monolith Conclusion
Contents
Our migration to microservices
$ rails new soundcloud How we started
Around 2012
Around 2012
Around 2012
Around 2012
After
After
But what about deployment?
Component Environment Zone Deployment Our abstractions
Deployment The process
The monolith
Track User Playlist The monolith Core entities
360 Chef provisioned bare-metal machines Rails 2.3 Capistrano deployment The
monolith The technology
The monolith The architecture Public API Internal API Public API
Strangler Internal API caching/ strangler Another BFF Many microservices Public Web Workers
The monolith The components Public API MoshiMoshi (Internal API) Public
Web Assets MoshiMoshi Comments (Internal API) Workers Cron Shell Migration
The monolith The hosts component statsd-exporter mtail statsd passenger- exporter
…
Utilization Deployment Lack of confidence The monolith The issues
Containerizing the monolith
Throw it into Kubernetes
Congrats, your monolith is a microservice now
Thank You!
1.5 Engineers 1 year until the last bit was cleaned
up The project
The first milestone
Docker development container Tests on GoCD The first milestone
The proof of concept
First staging component The proof of concept Does it even
work?
The proof of concept PR O BLEM ! Init script
Nginx Passenger Passenger Process Rails App ??? Where are my env variables?
The proof of concept Env variables with The Perl Hack™
SO LU TIO N
Productionizing
Deployment Monitoring Logs Productionizing Does it run where it matters?
Productionizing Anatomy of a traffic serving pod component statsd statsd-exporter
passenger- exporter mtail twemproxy twemproxy-cu twemproxy- exporter init
Don’t choke service discovery Be allocatable Productionizing Sizing the pods
CPU units for main container Passenger processes Productionizing Sizing the
pods 3 16
Productionizing PR O BLEM ! Stdout v/s the log metrics
exporter Component STDOUT Log aggregator File??? Mtail
Productionizing Mtail with The Rotatelogs Hack™ SO LU TIO N
Productionizing Public API
Orchestration Productionizing Public API
Productionizing Public API PR O BLEM ! DNS latency and
our excessive usage Component Service Service Service
Productionizing Public API CoreDNS and the DNS Hack™ SO LU
TIO N
Productionizing Internal API
Highest throughput Productionizing Internal API
Productionizing Internal API PR O BLEM ! Latency was too
high
Productionizing Internal API Optimize GC and make cheaper SQL queries
SO LU TIO N
Productionizing Internal API PR O BLEM ! Errors spikes during
deployment
Productionizing Internal API The preStop Trick™ SO LU TIO N
Productionizing Internal API PR O BLEM ! Errors spikes during
deployment (still???)
Productionizing Internal API The Pre Start Trick™ SO LU TIO
N
The rest
• Workers • Cron jobs • Shell / Migration hosts
• Cleanup! The rest
Finishing
Current status
On-prem Cloud Current status Number of pods ~1000 ~140
On-prem RPS Cloud RPS Current status Traffic 25K 3K
Current status Many deploys
Conclusions
One Infrastructure One Delivery Process Conclusions What we solved
Step by step Controlled rollouts Managing expectations Conclusions How we
did it
Improved utilization Increased confidence Enabling new initiatives Conclusions Benefits
Assess current progress Evaluate costs and benefits Conclusions Should you
do it?
Thank You!
@janogonzalez https://soundcloud.com/janogonzalez
More Decks by Jano González
See All by Jano González
Migrando a Microservicios
janogonzalez
1
370
Extracting services from a monolith
janogonzalez
3
320
¿Después de 10 años, realmente entiendo esta industria?
janogonzalez
3
560
Microservices in Practice
janogonzalez
7
740
Two programmers in one
janogonzalez
1
260
The Bipolar Programmer
janogonzalez
4
690
Ruby for your two internal programmers
janogonzalez
4
320
Ruby for Java minds
janogonzalez
4
1.2k
How Ruby Programmed Me
janogonzalez
11
580
Other Decks in Programming
See All in Programming
SODA - FACT BOOK(EN)
sodainc
1
250
AI時代のIaC入門 〜なぜインフラをコードで管理したいのか?〜 / Introduction to IaC in the AI Era
akihisaikeda
6
1.5k
Kawaiicon: Kubernetes, it's no longer cool, but let's secure it anyway!
barnbarn
0
130
止まらない比較を止める — 余帰納法で無限ストリームを比べる
tarao
0
190
動画プレイヤーはなぜカクつくのか 〜Flutterのレンダリングとネイティブの境界を解剖する〜
naruogram
0
340
우리의 Nitro Modules는 당신의 Turbo Modules보다 아름답다
l2hyunwoo
0
260
AI時代のシニアエンジニアとは
yosuke_furukawa
PRO
15
12k
あなたはMCPの機能を、まだ知らない。
4geru
1
400
선언형 햅틱 라이브러리 진동을 소개합니다
l2hyunwoo
1
320
作れるエンジニアから、価値を届けるエンジニアへ
ryunakayama
1
300
そのデザイントークン、Flutterでも同じ見た目になりますか?
nakamizu
0
370
将来の標準フォーマッター?ktfmtを深堀る
t45k
0
170
Featured
See All Featured
Self-Hosted WebAssembly Runtime for Runtime-Neutral Checkpoint/Restore in Edge–Cloud Continuum
chikuwait
0
950
Navigating Algorithm Shifts & AI Overviews - #SMXNext
aleyda
1
1.7k
More Than Pixels: Becoming A User Experience Designer
marktimemedia
3
620
End of SEO as We Know It (SMX Advanced Version)
ipullrank
3
4.5k
The Language of Interfaces
destraynor
162
27k
How to Get Subject Matter Experts Bought In and Actively Contributing to SEO & PR Initiatives.
livdayseo
1
240
Leading Effective Engineering Teams in the AI Era
addyosmani
9
2.9k
GraphQLとの向き合い方2022年版
quramy
50
15k
Lessons Learnt from Crawling 1000+ Websites
charlesmeaden
PRO
1
1.7k
How Software Deployment tools have changed in the past 20 years
geshan
3
35k
svc-hook: hooking system calls on ARM64 by binary rewriting
retrage
2
680
Getting science done with accelerated Python computing platforms
jacobtomlinson
2
570