Next generation infrastructure with Docker and Mesos

Inspired by Google, organizations are building out warehouse-scale infrastructure. In this model, applications packaged in lightweight containers run on large pools of computers to achieve high levels of flexibility and utilization. Web applications run on the same servers that are processing batch big data workloads and stream processing.

In this talk we'll cover two building blocks of this new infrastructure: Docker and Apache Mesos.

Docker allows organizations to easily package, run and share server applications. Since Docker's release over a year ago, it has become one of the fastest growing open source projects in history.

Mesos is a open-source distributed scheduler that is similar to the Omega scheduler that powers Google. Mesos is the original computation engine for Apache Spark, but has added the ability to run Docker containers, Hadoop and applications like Storm.

Transcript

1. D O C K E R + M E S O S S T L H A D O O P J U N E 2 0 1 4 S T E V E N B O R R E L L I @stevendborrelli steve@aster.is
2. A B O U T M E 2 0 1 4 F O U N D E D : O R G A N I Z E R O F S T L M A C H I N E L E A R N I N G A N D D O C K E R S T L F I R S T C O M P U T E R : @ S Y S T E M S E N G I N E E R I N G , H P C , B I G D A TA & C L O U D
3. Let’s talk about
4. I N F R A S T R U C T U R E T H E N E X T G E N E R A T I O N O F
5. Why should I care?
6. I N F R A S T R U C T U R E I S T H E F O U N D AT I O N O F T H E I N F O R M AT I O N E C O N O M Y
7. – M A R C A N D R E E S S E N “We are in the middle of a dramatic and broad technological and economic shift in which software companies are poised to take over large swathes of the economy.”
8. I N F R A S T R U C T U R E I S A C O M P E T I T I V E A D VA N TA G E
9. ƭ W R I T E T E S T D E P L O Y S P E E D + Q U A L I T Y O F S O F T WA R E D E V E L O P M E N T H A S B E C O M E C R I T I C A L
10. WHAT TRENDS ARE EMERGING?
11. WA R E H O U S E S C A L E C O M P U T I N G
12. I N S P I R E D B Y
13. GFS (2003) MapReduce (2004) Chubby (2006) BigTable (2006)
14. H O W D O E S G O O G L E R U N ?
15. APPLICATION CONTAINERS + DISTRIBUTED SCHEDULING H O W D O E S G O O G L E R U N ?
16. B U I L D I N G B L O C K S
17. C O N TA I N E R S PA C K A G E A P P L I C AT I O N S I N L I G H T W E I G H T V I RT U A L M A C H I N E S
18. D I S T R I B U T E D S C H E D U L I N G R U N W O R K L O A D S A C R O S S M U LT I P L E S E R V E R S
19. APPLICATION CONTAINERS + DISTRIBUTED SCHEDULING
20. Let’s start with containers
21. 1.0 released last week!
22. W H Y D O C K E R ? Docker makes it easy to: Package Deploy Share Containerized Server Applications
23. Think: java -jar vs. ./configure; make install !
24. What’s the difference between containers and virtual machines (VMs)? !
25. V I RT U A L I Z AT I O N
26. 1966-1972 IBM CP/CMS 1989 Insignia SoftPC 1997 Connectix VirtualPC 1999 VMWare Workstation 2001 IBM AIX LPAR 2002 Xen 2006 Amazon EC2 2007 Sun Logical Domains 2007 Linux KVM 2007 InnoTek VirtualBox 2008 Microsoft Hyper-V H A R D WA R E V I RT U A L I Z AT I O N
27. C O N TA I N E R S
28. P R O C E S S V I RT U A L I Z AT I O N 1979-1982 UNIX Chroot 1998 FreeBSD Jail 2001 Parallels Virtuozzo 2001 Linux-VServer 2005 Solaris Zones 2005 OpenVZ 2008 Linux LXC 2007+ PAAS: Heroku, Joyent, CloudFoundry 2013 Docker
29. (LINUX) KERNEL LAYER FS CGROUPS NAMESPACES LIBCONTAINER DOCKER LXC D O C K E R A R C H I T E C T U R E
30. N A M E S PA C E S V S . C G R O U P S Namespaces provide isolation: • pid (processes) • net (network interfaces, routing...) • ipc (System V IPC) • mnt (mount points, filesystems) • uts (hostname) • user (UIDs) Control groups control resources: • cpu (CPU shares) • cpusets (limit processes to a CPU) • memory (swap, dirty pages) • blockio (throttle reads/writes) • devices • net_cls, net_prio: control packet class and priority

31. Differences between containers and virtual machines ! • Weaker isolation in containers • Containers run near-native speed for CPU/IO • Cached containers launch in around 0.1 second (libcontainer) • Less memory overhead
32. EXAMPLES
33. R U N N I N G A C O N TA I N E R Start a container: Mount host filesystems:
34. Example: run Zookeeper + Exhibitor Host ports will be dynamically allocated by docker Port 2181 on host will be mapped to 2181 on container M A P P I N G P O RT S
35. New in 0.11, allows a container to access host adapters: D I R E C T H O S T N E T W O R K Port 8080 on the container is 8080 on the host:
36. It’s easy to build containers
37. J AVA D O C K E R F I L E
38. No we can build a Tomcat container from the Java container
39. T O M C AT D O C K E R F I L E
40. L AY E R E D F S I S A G R A P H
41. It’s easy to share containers
42. D O C K E R R E G I S T RY S E A R C H P U S H P U L L
43. D O C K E R S U M M A RY Easy to build, run & share containers Rapidly expanding ecosystem Better performance vs. VMs Layered filesystem gives us git-like control of images. Reduces complexity of system builds
44. We’ve packaged up all our apps into containers! ! Now how do we run them?
45. D I S T R I B U T E D S C H E D U L I N G
46. Launches 2 Billion containers/week ~3,300/second Powered by:
47. Can I implement Google-style computing?
50. B E R K E L E Y D ATA A N A LY T I C S S TA C K ( B D A S )
51. Fault Tolerant Massively Scalable Mix Diverse Workloads Efficient/Optimal Resource Utilization D E S I R E D C H A R A C T E R I S T I C S
52. S TAT I C PA R T I T I O N I N G C U R R E N T LY, A L M O S T E V E RY O N E U S E S
53. M I X W O R K L O A D S W I T H M E S O S
54. I N C R E A S E U T I L I Z AT I O N W I T H M E S O S
55. Mesos uses Zookeeper for High Availability
56. H I G H LY AVA I L A B L E A R C H I T E C T U R E
57. Mesos is a Two-Level Scheduler
58. Frameworks Manage Jobs Mesos Tracks and Provides Resources
59. R U N M U LT I P L E F R A M E W O R K S App-specific: Generic:
60. R U N M U LT I P L E F R A M E W O R K S
61. +
62. Package Apps into Containers Run Containers at Scale N AT U R A L C O M P L E M E N T S
63. M E S O S D O C K E R I N T E G R AT I O N Mesos Uses Cgroups for Isolation Fall 2013: External Script to shell out to Docker June 2014 0.19: Native Container Support Two Docker executors:
64. ! {! "cmd": "env && sleep 300",! "constraints": [! ["attribute", "OPERATOR", "value"]! ],! "container": {! "image": "docker:///dockeruser/oracle-java7",! "options": ["-v", "/etc/config:/etc/config:ro"]! },! "cpus": 2,! "env": {! "LD_LIBRARY_PATH": "/usr/local/lib/myLib"! }! } L A U N C H I N G D O C K E R C O N TA I N E R S W I T H M A R AT H O N
65. D O C K E R S U P P O RT F O R M E S O S Prototype Mesos Backend Planned Support
66. T O O L S C A M B R I A N E X P L O S I O N
67. Docker has a rapidly growing ecosystem Existing software is being rewritten to support Docker containers D O C K E R I S D R I V I N G N E W T O O L I N G
68. Docker-centric Big Data Pre-Docker PAAS D O C K E R S U P P O R T I N G I N F R A S T R U C T U R E
69. Service Discovery Dynamic Proxies Linking Containers Storage N E W M O D E L S E M E R G I N G
70. N E X T S T E P S Dockerize your applications, run them using simple scripts Use Mesos as a backend for Spark Start running apps via Marathon & Chronos Integrate containers into your build process
72. Docker/Mesos/AWS Professional Services ! Jenkins Plugin for Docker & Mesos ! Prebuilt Amazon AMI ! API to spin up clusters on AWS http://aster.is
73. I M A G E C R E D I T S • "U.S. Army Photo", number 163-12-62. Patsy Simmers, et. al holding computer parts: http://ftp.arl.army.mil/ ftp/historic-computers/ • “U.S. Army Photo”, Two women operating the ENIAC’s: http://ftp.arl.army.mil/ftp/historic-computers/png/ eniac7.png • Factory: http://commons.wikimedia.org/wiki/File:Float_Glass_Unloading.jpg • Distributed Scheduling: http://s1.reutersmedia.net/resources/r/? m=02&d=20120619&t=2&i=620575938&w=&fh=&fw=&ll=700&pl=390&r=2012-06-19T114519Z_02_GM1E86I1 B6M01_RTRRPP_0_CHINA • Mesos Paper: http://static.usenix.org/event/nsdi11/tech/full_papers/Hindman_new.pdf • Static Workloads/Mix Workloads/Mix Utilization, Ben Hindman, http://www.slideshare.net/dotCloud/high- speed-shipping-lanes-how-containers-are-revolutionizing-distributed-computing-at-scale • Cambrian Explosion: http://www.bio.miami.edu/dana/pix/cambrian_explosion.jpg