Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How we learned to love the Data Center Operating System
Search
DevOps Pro
June 01, 2016
Technology
0
150
How we learned to love the Data Center Operating System
by Saulius Valatka
DevOps Pro Vilnius 2016
DevOps Pro
June 01, 2016
Tweet
Share
More Decks by DevOps Pro
See All by DevOps Pro
Practical Application Logging and Monitoring
devopspro
0
100
Google Cloud Platform for DevOps Professionals
devopspro
0
280
Dev vs Ops transformation to Agility DevOps culture
devopspro
1
230
Using Test Automation for Continuous Integration, Delivery and Quality
devopspro
0
86
How the hell do I run my microservices in production, and will it scale?
devopspro
0
120
Accelerating innovation through AWS Cloud technology
devopspro
0
87
The Field Guide to Understanding Declarative Systems
devopspro
0
80
DevOps is the answer! What was the question again?
devopspro
0
150
From Monolith to Docker Distributed Applications
devopspro
0
240
Other Decks in Technology
See All in Technology
今年のRubyKaigiはProfiler Year🤘
osyoyu
0
520
Babylon.jsと色々なものを組み合わせる:ブラウザのAPIやガジェットや2D描画ライブラリなど / Babylon.js 勉強会 vol.3
you
PRO
0
190
EM完全に理解した と思ったけど、 やっぱり何も分からなかった話 / EM Night Fukuoka #1
hirutas
0
310
中年男性がメインフレームから クラウドへキャリアシフトしてみた
uechishingo
1
420
データベース03: 関係データモデル
trycycle
0
110
コードや知識を組み込む / Incorporate Code and knowledge
ks91
PRO
0
160
.NET Profiler in 2024.
kkamegawa
2
2.5k
kcp: Kubernetes APIs Are All You Need #techfeed_live / TechFeed Experts Night 28th
ytaka23
1
160
Cypress or Playwright?
rainerhahnekamp
0
180
高専で制御を、大学でセンシングを学び、次は脳みそ
satoshirobatofujimoto
0
120
個人的、Kubernetes の最新注目機能! (2024年5月版) / TechFeed Experts Night#28 〜 コンテナ技術最前線
pfn
PRO
1
130
How to do well in consulting–Balkan Ruby 2024
irinanazarova
0
180
Featured
See All Featured
The Cost Of JavaScript in 2023
addyosmani
21
3.9k
Learning to Love Humans: Emotional Interface Design
aarron
267
39k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
221
21k
Music & Morning Musume
bryan
41
5.6k
Automating Front-end Workflow
addyosmani
1357
200k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
15
1.6k
Faster Mobile Websites
deanohume
300
30k
Done Done
chrislema
178
15k
Building Applications with DynamoDB
mza
88
5.6k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
80
44k
How GitHub Uses GitHub to Build GitHub
holman
468
290k
A Tale of Four Properties
chriscoyier
153
22k
Transcript
HOW WE LEARNED TO LOVE THE DATA CENTER OPERATING SYSTEM
SAULIUS VALATKA / ADFORM
Online Advertising Full Stack Platform
Online Advertising Full Stack Platform Realtime “smart“ ads Forecasting, fraud
detection, etc.
Online Advertising Full Stack Platform Realtime “smart“ ads Forecasting, fraud
detection, etc. 1mln QPS under 100ms 1TB daily data
EVOLUTION
MIDDLE AGES ctrtrain.ec2-aws.com test2.ec2-aws.com modelling.ec2-aws.com
THE TORTURE # yum install python R libboost-3.12 $ scp
script.R test.aws.com:/script.R # crontab -e
THE TORTURE # yum install python R libboost-3.12 $ scp
script.R test.aws.com:/script.R # crontab -e “strange, worked on my machine …”
RENAISSANCE ctrtrain.ec2-aws.com test2.ec2-aws.com worker-1.adform.com worker-2.adform.com ab34na3n ar2afga3n
None
CONTAINERIZE ! self contained artifacts isolated runtime basically no overhead
unified deployment
BUT WAIT … what about configuration ?
The twelve-factor app stores config in environment variables Env vars
are easy to change between deploys without changing any code There is little chance of them being checked into the code repo accidentally They are a language- and OS-agnostic standard
BUT WAIT … where do I log ? and what
about metrics ?
None
MODERN ERA e34sadf ab34na3n af4f5a4r aafde33a fa45daws faes4fa3 aaf444a2 fas3rfa4
None
MARATHON the init of the DCOS constraints deployment { "id":
“my-nginx", "container": { "type": "DOCKER", "docker": { "image": "nginx:1.7.7", "network": "BRIDGE", } }, "instances": 1, "cpus": 0.5, "mem": 128 }
SPRINT the exec of he DCOS will be open sourced
scheduler to follow!
MANAGING RESOURCES how much memory do I really need ?
MANAGING RESOURCES how much memory do I really need ?
and CPUs ? what does 0.5 CPUs mean anyway ?
MANAGING RESOURCES how much memory do I really need ?
and CPUs ? what does 0.5 CPUs mean anyway ? and what happens with the network ?
None
ISOLATION cgroups: cpu cpuset memory blkio net_cls
NETWORK ISOLATION Layer 3 routing software defined networks
ATOMIC AGE a4faw3f 4afsdgg asdf4faf se4faw aw3d3ff g4aefgsd 5gsdgr54s a4rff4afa
4f4qaf4
SERVICE DISCOVERY where is my app ? how do I
reach it ? won’t containers conflict about ports ?
MARATHON-LB
PERSISTENCE so .. where do I store my data ?
on the host ? won’t it disappear ?
PERSISTENCE / /opt/app/cache /var/lib/docker/devicemapper /var/lib/mesos/slave/volumes
PERSISTENCE / /opt/app/cache /opt/app/profile /var/lib/docker/devicemapper /var/lib/mesos/slave/volumes /mnt/sdc network block storage
FUTURE PLANS DC/OS IP per container Containerize all the things
@adforminsider