Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How we learned to love the Data Center Operating System
Search
DevOps Pro
June 01, 2016
Technology
0
160
How we learned to love the Data Center Operating System
by Saulius Valatka
DevOps Pro Vilnius 2016
DevOps Pro
June 01, 2016
Tweet
Share
More Decks by DevOps Pro
See All by DevOps Pro
Practical Application Logging and Monitoring
devopspro
0
100
Google Cloud Platform for DevOps Professionals
devopspro
0
280
Dev vs Ops transformation to Agility DevOps culture
devopspro
1
230
Using Test Automation for Continuous Integration, Delivery and Quality
devopspro
0
87
How the hell do I run my microservices in production, and will it scale?
devopspro
0
120
Accelerating innovation through AWS Cloud technology
devopspro
0
87
The Field Guide to Understanding Declarative Systems
devopspro
0
80
DevOps is the answer! What was the question again?
devopspro
0
150
From Monolith to Docker Distributed Applications
devopspro
0
250
Other Decks in Technology
See All in Technology
エンジニアの生存戦略 〜クラウド潮流の経験から紐解く技術トレンドのメカニズムと乗りこなし方〜
shimy
9
1.9k
Azure OpenAI Service Dev Day / LLMでできる!使える!生成AIエージェント
masahiro_nishimi
3
750
頼られるのが大好きな 皆さんへ - 支援相手との期待の合わせ方、突き放し方 -/For_people_who_like_to_be_relied_on
naitosatoshi
1
290
Docker互換のセキュアなコンテナ実行環境「Podman」超入門
devops_vtj
6
3.2k
JBUG岡山 #6 WordCamp男木島の チームビルディング
takeshifurusato
0
150
エンジニア向け会社紹介資料
caddi_eng
14
220k
[NIKKEI Tech Talk]Bias for Action!! 実践から学ぶための仕組とコミュニティ / Community for Practice and Learning
kanamasa
0
270
フルリモートワークはエンジニアの夢を叶えたか? #cm_odyssey
mamohacy
2
600
AWSでRAGを作る法方
sonoda_mj
1
140
目標設定は好きですか? アジャイルとともに目標と向き合い続ける方法 / Do you like target Management?
kakehashi
10
3k
Flutter研修【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
160
サービスの持続的な成長と技術負債について
siva_official
PRO
10
4.4k
Featured
See All Featured
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
35
6.3k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
24
1.8k
Learning to Love Humans: Emotional Interface Design
aarron
269
39k
Git: the NoSQL Database
bkeepers
PRO
423
64k
Building a Scalable Design System with Sketch
lauravandoore
458
32k
No one is an island. Learnings from fostering a developers community.
thoeni
17
2.8k
From Idea to $5000 a Month in 5 Months
shpigford
377
46k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
228
16k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
26
2.1k
Speed Design
sergeychernyshev
9
270
VelocityConf: Rendering Performance Case Studies
addyosmani
321
23k
Mobile First: as difficult as doing things right
swwweet
219
8.8k
Transcript
HOW WE LEARNED TO LOVE THE DATA CENTER OPERATING SYSTEM
SAULIUS VALATKA / ADFORM
Online Advertising Full Stack Platform
Online Advertising Full Stack Platform Realtime “smart“ ads Forecasting, fraud
detection, etc.
Online Advertising Full Stack Platform Realtime “smart“ ads Forecasting, fraud
detection, etc. 1mln QPS under 100ms 1TB daily data
EVOLUTION
MIDDLE AGES ctrtrain.ec2-aws.com test2.ec2-aws.com modelling.ec2-aws.com
THE TORTURE # yum install python R libboost-3.12 $ scp
script.R test.aws.com:/script.R # crontab -e
THE TORTURE # yum install python R libboost-3.12 $ scp
script.R test.aws.com:/script.R # crontab -e “strange, worked on my machine …”
RENAISSANCE ctrtrain.ec2-aws.com test2.ec2-aws.com worker-1.adform.com worker-2.adform.com ab34na3n ar2afga3n
None
CONTAINERIZE ! self contained artifacts isolated runtime basically no overhead
unified deployment
BUT WAIT … what about configuration ?
The twelve-factor app stores config in environment variables Env vars
are easy to change between deploys without changing any code There is little chance of them being checked into the code repo accidentally They are a language- and OS-agnostic standard
BUT WAIT … where do I log ? and what
about metrics ?
None
MODERN ERA e34sadf ab34na3n af4f5a4r aafde33a fa45daws faes4fa3 aaf444a2 fas3rfa4
None
MARATHON the init of the DCOS constraints deployment { "id":
“my-nginx", "container": { "type": "DOCKER", "docker": { "image": "nginx:1.7.7", "network": "BRIDGE", } }, "instances": 1, "cpus": 0.5, "mem": 128 }
SPRINT the exec of he DCOS will be open sourced
scheduler to follow!
MANAGING RESOURCES how much memory do I really need ?
MANAGING RESOURCES how much memory do I really need ?
and CPUs ? what does 0.5 CPUs mean anyway ?
MANAGING RESOURCES how much memory do I really need ?
and CPUs ? what does 0.5 CPUs mean anyway ? and what happens with the network ?
None
ISOLATION cgroups: cpu cpuset memory blkio net_cls
NETWORK ISOLATION Layer 3 routing software defined networks
ATOMIC AGE a4faw3f 4afsdgg asdf4faf se4faw aw3d3ff g4aefgsd 5gsdgr54s a4rff4afa
4f4qaf4
SERVICE DISCOVERY where is my app ? how do I
reach it ? won’t containers conflict about ports ?
MARATHON-LB
PERSISTENCE so .. where do I store my data ?
on the host ? won’t it disappear ?
PERSISTENCE / /opt/app/cache /var/lib/docker/devicemapper /var/lib/mesos/slave/volumes
PERSISTENCE / /opt/app/cache /opt/app/profile /var/lib/docker/devicemapper /var/lib/mesos/slave/volumes /mnt/sdc network block storage
FUTURE PLANS DC/OS IP per container Containerize all the things
@adforminsider