How we learned to love the Data Center Operating System

How we learned to love the Data Center Operating System

by Saulius Valatka
DevOps Pro Vilnius 2016

6d46284ef16436cb154adf4963e236f0?s=128

DevOps Pro

June 01, 2016
Tweet

Transcript

  1. HOW WE LEARNED TO LOVE THE DATA CENTER OPERATING SYSTEM

    SAULIUS VALATKA / ADFORM
  2. Online Advertising Full Stack Platform

  3. Online Advertising Full Stack Platform Realtime “smart“ ads Forecasting, fraud

    detection, etc.
  4. Online Advertising Full Stack Platform Realtime “smart“ ads Forecasting, fraud

    detection, etc. 1mln QPS under 100ms 1TB daily data
  5. EVOLUTION

  6. MIDDLE AGES ctrtrain.ec2-aws.com test2.ec2-aws.com modelling.ec2-aws.com

  7. THE TORTURE # yum install python R libboost-3.12 $ scp

    script.R test.aws.com:/script.R # crontab -e
  8. THE TORTURE # yum install python R libboost-3.12 $ scp

    script.R test.aws.com:/script.R # crontab -e “strange, worked on my machine …”
  9. RENAISSANCE ctrtrain.ec2-aws.com test2.ec2-aws.com worker-1.adform.com worker-2.adform.com ab34na3n ar2afga3n

  10. None
  11. CONTAINERIZE ! self contained artifacts isolated runtime basically no overhead

    unified deployment
  12. BUT WAIT … what about configuration ?

  13. The twelve-factor app stores config in environment variables Env vars

    are easy to change between deploys without changing any code There is little chance of them being checked into the code repo accidentally They are a language- and OS-agnostic standard
  14. BUT WAIT … where do I log ? and what

    about metrics ?
  15. None
  16. MODERN ERA e34sadf ab34na3n af4f5a4r aafde33a fa45daws faes4fa3 aaf444a2 fas3rfa4

  17. None
  18. MARATHON the init of the DCOS constraints deployment { "id":

    “my-nginx", "container": { "type": "DOCKER", "docker": { "image": "nginx:1.7.7", "network": "BRIDGE", } }, "instances": 1, "cpus": 0.5, "mem": 128 }
  19. SPRINT the exec of he DCOS will be open sourced

    scheduler to follow!
  20. MANAGING RESOURCES how much memory do I really need ?

  21. MANAGING RESOURCES how much memory do I really need ?

    and CPUs ? what does 0.5 CPUs mean anyway ?
  22. MANAGING RESOURCES how much memory do I really need ?

    and CPUs ? what does 0.5 CPUs mean anyway ? and what happens with the network ?
  23. None
  24. ISOLATION cgroups: cpu cpuset memory blkio net_cls

  25. NETWORK ISOLATION Layer 3 routing software defined networks

  26. ATOMIC AGE a4faw3f 4afsdgg asdf4faf se4faw aw3d3ff g4aefgsd 5gsdgr54s a4rff4afa

    4f4qaf4
  27. SERVICE DISCOVERY where is my app ? how do I

    reach it ? won’t containers conflict about ports ?
  28. MARATHON-LB

  29. PERSISTENCE so .. where do I store my data ?

    on the host ? won’t it disappear ?
  30. PERSISTENCE / /opt/app/cache /var/lib/docker/devicemapper /var/lib/mesos/slave/volumes

  31. PERSISTENCE / /opt/app/cache /opt/app/profile /var/lib/docker/devicemapper /var/lib/mesos/slave/volumes /mnt/sdc network block storage

  32. FUTURE PLANS DC/OS IP per container Containerize all the things

  33. @adforminsider