Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
How we learned to love the Data Center Operating System
DevOps Pro
June 01, 2016
Technology
0
130
How we learned to love the Data Center Operating System
by Saulius Valatka
DevOps Pro Vilnius 2016
DevOps Pro
June 01, 2016
Tweet
Share
More Decks by DevOps Pro
See All by DevOps Pro
Practical Application Logging and Monitoring
devopspro
0
67
Google Cloud Platform for DevOps Professionals
devopspro
0
240
Dev vs Ops transformation to Agility DevOps culture
devopspro
1
220
Using Test Automation for Continuous Integration, Delivery and Quality
devopspro
0
77
How the hell do I run my microservices in production, and will it scale?
devopspro
0
110
Accelerating innovation through AWS Cloud technology
devopspro
0
74
The Field Guide to Understanding Declarative Systems
devopspro
0
69
DevOps is the answer! What was the question again?
devopspro
0
140
From Monolith to Docker Distributed Applications
devopspro
0
230
Other Decks in Technology
See All in Technology
【toranoana.deno#7】Denoからwasmを呼び出す基礎
toranoana
0
100
Modern Android dependency injection
hugovisser
1
120
Azure Arc Virtual MachineとAzure Arc Resource Bridge / VM provisioning through Azure portal on Azure Stack HCI (preview)
sashizaki
0
120
Microsoft Build 2022 Recap Party!! Azure のデータ & 分析サービス 注目アップデート / microsoft-build-2022-recap-azure-data-and-analytics
nakazax
0
250
アジャイル推進活動におけるBeAgileへの変化の兆し/Signs_of_Change_to_"Be_Agile"_in_Agile_Promotion_Activities
m_iyama
0
150
データ分析で切り拓け! エンジニアとしてのデータ分析職キャリア戦略
ksnt
0
110
JAWS-UG re:Habilitaion 報告 / JAWS-UG OITA rehabilitation
hiranofumio
0
120
QiitaConference2022
fuwasegu
0
160
ひとりでも安定して 組織を変える活動を続けていくための ストレスマネジメント
pastelinc
0
810
UWBを使ってみた
norioikedo
0
400
ラブグラフ紹介資料 〜プロダクト解体新書〜 / Lovegraph Product Deck
lovegraph
0
160
ノーメンテナンス運用実現のためのノウハウ/ColoplTech-05-02
colopl
0
160
Featured
See All Featured
Atom: Resistance is Futile
akmur
255
20k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
39
13k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
181
15k
Keith and Marios Guide to Fast Websites
keithpitt
404
21k
Building Applications with DynamoDB
mza
83
4.7k
It's Worth the Effort
3n
172
25k
The Pragmatic Product Professional
lauravandoore
19
3k
Designing on Purpose - Digital PM Summit 2013
jponch
106
5.6k
YesSQL, Process and Tooling at Scale
rocio
157
12k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
29
4.3k
How GitHub (no longer) Works
holman
296
140k
A Tale of Four Properties
chriscoyier
149
21k
Transcript
HOW WE LEARNED TO LOVE THE DATA CENTER OPERATING SYSTEM
SAULIUS VALATKA / ADFORM
Online Advertising Full Stack Platform
Online Advertising Full Stack Platform Realtime “smart“ ads Forecasting, fraud
detection, etc.
Online Advertising Full Stack Platform Realtime “smart“ ads Forecasting, fraud
detection, etc. 1mln QPS under 100ms 1TB daily data
EVOLUTION
MIDDLE AGES ctrtrain.ec2-aws.com test2.ec2-aws.com modelling.ec2-aws.com
THE TORTURE # yum install python R libboost-3.12 $ scp
script.R test.aws.com:/script.R # crontab -e
THE TORTURE # yum install python R libboost-3.12 $ scp
script.R test.aws.com:/script.R # crontab -e “strange, worked on my machine …”
RENAISSANCE ctrtrain.ec2-aws.com test2.ec2-aws.com worker-1.adform.com worker-2.adform.com ab34na3n ar2afga3n
None
CONTAINERIZE ! self contained artifacts isolated runtime basically no overhead
unified deployment
BUT WAIT … what about configuration ?
The twelve-factor app stores config in environment variables Env vars
are easy to change between deploys without changing any code There is little chance of them being checked into the code repo accidentally They are a language- and OS-agnostic standard
BUT WAIT … where do I log ? and what
about metrics ?
None
MODERN ERA e34sadf ab34na3n af4f5a4r aafde33a fa45daws faes4fa3 aaf444a2 fas3rfa4
None
MARATHON the init of the DCOS constraints deployment { "id":
“my-nginx", "container": { "type": "DOCKER", "docker": { "image": "nginx:1.7.7", "network": "BRIDGE", } }, "instances": 1, "cpus": 0.5, "mem": 128 }
SPRINT the exec of he DCOS will be open sourced
scheduler to follow!
MANAGING RESOURCES how much memory do I really need ?
MANAGING RESOURCES how much memory do I really need ?
and CPUs ? what does 0.5 CPUs mean anyway ?
MANAGING RESOURCES how much memory do I really need ?
and CPUs ? what does 0.5 CPUs mean anyway ? and what happens with the network ?
None
ISOLATION cgroups: cpu cpuset memory blkio net_cls
NETWORK ISOLATION Layer 3 routing software defined networks
ATOMIC AGE a4faw3f 4afsdgg asdf4faf se4faw aw3d3ff g4aefgsd 5gsdgr54s a4rff4afa
4f4qaf4
SERVICE DISCOVERY where is my app ? how do I
reach it ? won’t containers conflict about ports ?
MARATHON-LB
PERSISTENCE so .. where do I store my data ?
on the host ? won’t it disappear ?
PERSISTENCE / /opt/app/cache /var/lib/docker/devicemapper /var/lib/mesos/slave/volumes
PERSISTENCE / /opt/app/cache /opt/app/profile /var/lib/docker/devicemapper /var/lib/mesos/slave/volumes /mnt/sdc network block storage
FUTURE PLANS DC/OS IP per container Containerize all the things
@adforminsider