RingPlus: My God... It's Full of Containers

A description of Docker / Linux containers and their applications and drawbacks in relation to realtime communications systems.

Originally presented at ClueCon 2015

Transcript

1. My God… It’s Full of Containers
2. MVNO/MVNE running on the Sprint Network Stack -FreeSWITCH, Adhearsion, PostgreSQL, Kamailio Redis, IMS, KVM, Docker, etc… Business is consumer wireless disruption
3. Containers - The Whats
4. Hardware Level Virtualization vs OS Level Virtualization Virtual Machines vs Containers
5. Virtual Machines vs Containers • Virtual Machines • Provide a hardware abstraction layer • Complete OS • Slower startup • Dev is separate from Prod • Additional performance overhead • Containers • LXC has been around for a decade, OpenVZ • Can be full systems with init, persistent storage, • Can be shared read-only bins/libs with just the application • Lightweight and distributable • Packing density due to shared components
6. Kittens or Cattle • VMs are kittens (bare metal is more like a shark, but that’s a different talk.) We give them names, hand feed them with private Chefs, and care about their wellbeing.
 • Containers are like cattle. They have numbers and are managed in herds. They are replaceable and no one will notice.
7. Enter the Docker
8. • Opinionated Containers (if you like Rails, you might like Docker) • Containers should be single-process, no durable storage, ephemeral little creatures • Filesystems are built of read-only layers using AuFS - only the top layer is read/write • It shares underlying libs/bins so the docker image can be tiny-ish.
 
 
 • Networking using an ethernet bridge - `docker0.` • By default each container is given an 172.17.xxx.xxx/16 inside the bridge and specific ports are mapped via NAT Docker
9. INTERESTING BITS FOR RTC
10. Interesting RTC Bits KVM loses by 4% - Who Cares! Everybody Wins! KVM vs Docker vs Native 0" 5" 10" 15" 20" 25" 30" 35" 40" KVM" Docker" Na3ve" sysbench --test=cpu --cpu-max-prime=20000 run
11. Interesting RTC Bits Disk IO - Not Really An Issue • AuFS • Layered filesystem is efficient and performant enough for most tasks • Better IO needed? Easy: • Data-only volume containers. Bypasses AuFS, just uses standard disk. • Need to share that data elsewhere? Spin NFS container process to share between hosts also mounts that container. • Each process can mount it separately.
12. Interesting RTC Bits Networking… Networking… Networking…
13. A Tangent on Docker Networking and its Horribleness • Ethernet Bridge Nat is/can be *terrible* • YMMV - up to 50% performance decrease (<1.7) • Even more important - inter-host communication is a no-no. • All containers are placed into a 172.17.xxx.xxx /16 address space - unreachable from network hosts • Solutions: Overlay networks • Current solutions: • Userspace Processing — options like Weave can drop throughput by 90% as all packets are sent through a userspace daemon on both sides. • Proposed solution: VXLANs in-kernel forwarding (Flannel-esque) or networking plugins. Seems reasonable. Isn’t standard yet. • You VM users - update your libvirt! Latency in test systems went from 20-30ms per hop to ~4ms on CentOS libvirt upgrades.
14. A Tangent on Docker Networking and its Horribleness • For any RTC, you MUST use net=host networking • Exposes the native networking stack within the container • Side Affect: Horribly insecure - ordering a shutdown command will terminate the entire host. • https://github.com/docker/docker/issues/6401 • Requires setting physical host network appropriately (multiple virtual IPs) • Provides near native-level network throughput at high sustained PPS • For much further info, see this IBM Research Paper http://domino.research.ibm.com/library/cyberdig.nsf/papers/ 0929052195DD819C85257D2300681E7B/$File/rc25482.pdf
15. Docker
16. SO LET US THINK LIKE A DOCKER
17. Kamailio RTPProxy Data Volume Postgres NFS FreeSWITCH Adhearsion Voicemails, IVRs S3/NFS • A set of physical or virtual machines • FreeSWITCH/Kamailio aren’t blockers • Adhearsion & RTPProxy • Service Discovery to determine new components • Components can advertise services and announce locations, ports, etc.. • Orchestration layer can monitor cattle and restart where/when appropriate Service Discovery Kubernetes / etcd / Compose Examination of an RTC Container Stack API
18. Building FS on Docker FROM debian:latest MAINTAINER Evan McGee <evan@ringplus.net> # Install FreeSWITCH RUN apt-get update && apt-get install -y wget RUN echo "deb http://files.freeswitch.org/repo/deb/debian/ jessie main" > /etc/apt/sources.list.d/freeswitch.list RUN wget -O - http://files.freeswitch.org/repo/deb/debian/key.gpg | apt-key add - RUN apt-get update && \ apt-get -y upgrade && \ DEBIAN_FRONTEND=none APT_LISTCHANGES_FRONTEND=none apt-get install -y --force-yes freeswitch-video-deps-most locales debconf RUN locale-gen en_US en_US.UTF-8 RUN cd /usr/src && git clone https://freeswitch.org/stash/scm/fs/freeswitch.git RUN cd /usr/src/freeswitch && ./bootstrap.sh -j && ./configure -C RUN cd /usr/src/freeswitch && perl -i -pe 's/#applications\/mod_av/applications\/mod_av/g' modules.conf && make && make install COPY certs /usr/local/freeswitch/certs RUN rm -rf /usr/src/freeswitch COPY conf /usr/local/freeswitch CMD ["/usr/local/freeswitch/bin/freeswitch", "-c", "-nonat"] docker build -t freeswitch_v1.6 .
19. Building FS on Docker REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE freeswitch_v1.6 latest 9d146660f9bf 32 minutes ago 2.173 GB <none> <none> cb768f6c43e3 46 minutes ago 1.538 GB freeswitch_v1.4 latest fcc538c9197f 9 weeks ago 1.612 GB <none> <none> c228a3df92f6 9 weeks ago 136.2 MB <none> <none> 5e3296728da0 9 weeks ago 1.141 GB kamailio latest f98ff4c0074a 15 weeks ago 330.9 MB rtpproxy latest 9672f76f8db8 15 weeks ago 423 MB <none> <none> 58590b2be6e6 15 weeks ago 519.9 MB debian stable fee16a8b35ef 20 weeks ago 125.2 MB debian latest df2a0347c9d0 20 weeks ago 125.2 MB
20. • Installing Docker • Ubuntu / CentOS / Jessie Backports packages (among many others) • curl -sSL https://get.docker.com/ | sh
 • Pull images from Docker Hub • https://hub.docker.com/ • Remember to not think in terms of a virtual machine, think in terms of shipping applications • Redo your Ansible / Chef / Puppet deploys. • Once you’ve got a grasp of container deployment, check out Google’s Kubernetes as an orchestration platform. • Provides a partial networking solution, scaleable deployment paradigm and ensures a cattle mentality.
21. TL;DR • Containers are great if you’re deploying a VMs that unnecessarily duplicate services or libraries • Containers are great if you’re in love density/performance/scaling ( do you also love NodeJS? ) • Containers are great at running in development exactly what runs on production
 
 • Containers are great if you love working with your network stack • Containers are great if you want to redo your deployment architecture • Containers are great if you love discovering something new
22. Final Thoughts • Combine and Conquer - why not run a lightweight VM to abstract away all hardware and then ship applications via containers inside? • If this is interesting, checkout out CoreOS and rkt. They dial containers up to 11. • https://www.digitalocean.com/community/tutorials/the-docker-ecosystem-an-introduction- to-common-components
23. Evan McGee @startledmarmot 
 evan@ringplus.net