Containers and Microservices make performance worse

by Gareth Rushgrove

Slide 1

Slide 1 text

Containers and Microservices Puppet Labs Gareth Rushgrove Make everything performance worse

Slide 2

Slide 2 text

Gareth Rushgrove @garethr

Slide 3

Slide 3 text

Gareth Rushgrove

Slide 4

Slide 4 text

Gareth Rushgrove

Slide 5

Slide 5 text

Quiz Time

Slide 6

Slide 6 text

Gareth Rushgrove Apologies to the following. I love you really.

Slide 7

Slide 7 text

This Talk

Slide 8

Slide 8 text

Gareth Rushgrove

Slide 9

Slide 9 text

make snide and sharply critical comments Gareth Rushgrove snark /sna:k/ noun

Slide 10

Slide 10 text

Fast in-process communication vs slow and unreliable HTTP Gareth Rushgrove

Slide 11

Slide 11 text

The performance cost of an overlay network Gareth Rushgrove

Slide 12

Slide 12 text

What happened to ps and friends? Gareth Rushgrove

Slide 13

Slide 13 text

Microservices and Performance It will get bad before it gets good

Slide 14

Slide 14 text

Monoliths are pretty easy to understand at a high level Gareth Rushgrove

Slide 15

Slide 15 text

Gareth Rushgrove Monolith

Slide 16

Slide 16 text

Even adding in the supporting infrastructure it’s not so bad Gareth Rushgrove

Slide 17

Slide 17 text

Gareth Rushgrove Monolith Database Load Balancer Network Network

Slide 18

Slide 18 text

I can optimise the network, the load balancer, the database, the application or the client. Easy Gareth Rushgrove

Slide 19

Slide 19 text

What about Microservices? Well, ﬁrst start with a diagram like this… Gareth Rushgrove

Slide 20

Slide 20 text

Microservice Microservice Microservice Microservice Microservice Microservice Microservice Microservice Gareth Rushgrove

Slide 21

Slide 21 text

Whatever protocol you’re using lets assume you’re communicating over the network… Gareth Rushgrove

Slide 22

Slide 22 text

Microservice Microservice Microservice Microservice Microservice Microservice Microservice Microservice Gareth Rushgrove Network Network Network Network Network Network Network Network

Slide 23

Slide 23 text

And you probably want more than one instance of each service… Gareth Rushgrove

Slide 24

Slide 24 text

Gareth Rushgrove Microservice Load Balancer Network

Slide 25

Slide 25 text

Gareth Rushgrove Microservice Microservice Microservice Microservice Microservice Microservice Microservice Microservice Network Network Network Network Network Network Network Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network

Slide 26

Slide 26 text

And you probably have a few different databases now too… Gareth Rushgrove

Slide 27

Slide 27 text

Slide 28

Slide 28 text

In my made-up 8 service architecture we went from 5 things to optimise up to 32 Gareth Rushgrove The Bad

Slide 29

Slide 29 text

We went from 3 network hops to, er, more depending on the request Gareth Rushgrove The Bad

Slide 30

Slide 30 text

We ignored the cost of serialisation/deserialisation (JSON can be expensive) Gareth Rushgrove The Bad

Slide 31

Slide 31 text

The operational overhead just jumped considerably Gareth Rushgrove The Bad

Slide 32

Slide 32 text

Lots more network trafﬁc. Watch out for latency in particular Gareth Rushgrove The Bad

Slide 33

Slide 33 text

Without request tracing you’re doomed Gareth Rushgrove The Bad

Slide 34

Slide 34 text

Granular services are easier to optimise individually Gareth Rushgrove The Good

Slide 35

Slide 35 text

Individual services can be scaled independently Gareth Rushgrove The Good

Slide 36

Slide 36 text

Debugging with Containers Is that process inside or outside a container?

Slide 37

Slide 37 text

Problems with free and top Gareth Rushgrove

Slide 38

Slide 38 text

Gareth Rushgrove $ free total used free shared buffers cached Mem: 1024444 864140 160304 5024 50008 637736 -/+ buffers/cache: 176396 848048 Swap: 473084 16 473068 $ docker exec test-container free total used free shared buffers cached Mem: 1024444 866440 158004 5024 50000 637732 -/+ buffers/cache: 178708 845736 Swap: 473084 16 473068 Can a container use that memory?

Slide 39

Slide 39 text

memory stats come from the proc ﬁlesystem: /proc/meminfo, / proc/vmstat, etc. Gareth Rushgrove

Slide 40

Slide 40 text

/proc/meminfo and /proc/vmstat are not aware of cgroups Gareth Rushgrove

Slide 41

Slide 41 text

Problems with ps Gareth Rushgrove

Slide 42

Slide 42 text

Gareth Rushgrove $ ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND ... 999 1807 0.2 11.4 867624 464572 ? Ssl 09:38 0:21 mysqld Is this process in a container?

Slide 43

Slide 43 text

Gareth Rushgrove $ ps -eo ucmd,cgroup COMMAND CGROUP ... mysqld 9:perf_event:/docker/61e76d2c39121282474ff895b9b3ba2addd775cdea6d2ba89ce76c28 Which container is that?

Slide 44

Slide 44 text

Gareth Rushgrove Sysdig

Slide 45

Slide 45 text

Provides a Kernel module, which hooks into cgroups and namespaces Gareth Rushgrove

Slide 46

Slide 46 text

Gareth Rushgrove $ sudo sysdig -c topcontainers_cpu CPU% container.name ----------------------------------------------------------------------- 90.13% mysql 15.93% wordpress1 7.27% haproxy 3.46% wordpress2 CPU usage across containers

Slide 47

Slide 47 text

Gareth Rushgrove $ sudo sysdig -pc -c topprocs_cpu container.name=client CPU% Process container.name ---------------------------------------------- 02.69% bash client 31.04% curl client 0.74% sleep client CPU usage in a single container

Slide 48

Slide 48 text

Gareth Rushgrove $ sudo sysdig -pc -c topprocs_net Bytes Process Host_pid Container_pid container.name --------------------------------------------------------------- 72.06KB haproxy 7385 13 haproxy 56.96KB docker.io 1775 7039 host 44.45KB mysqld 6995 91 mysql 44.45KB mysqld 6995 99 mysql 29.36KB apache2 7893 124 wordpress1 29.36KB apache2 26895 126 wordpress4 29.36KB apache2 26622 131 wordpress2 29.36KB apache2 27935 132 wordpress3 29.36KB apache2 27306 125 wordpress4 22.23KB mysqld 6995 90 mysqlclient Network bandwidth

Slide 49

Slide 49 text

Gareth Rushgrove $ sudo sysdig -pc -A -c echo_fds "fd.ip=172.17.0.3 and fd.ip=172.17.0.7" ------ Write 103B to [haproxy] [d468ee81543a] 172.17.0.7:37557->172.17.0.3:80 (hapr GET / HTTP/1.1 User-Agent: curl/7.35.0 Host: 172.17.0.7 Accept: */* X-Forwarded-For: 172.17.0.8 ------ Read 103B from [wordpress1] [12b8c6a04031] 172.17.0.7:37557->172.17.0.3:80 ( GET / HTTP/1.1 User-Agent: curl/7.35.0 Host: 172.17.0.7 Accept: */* X-Forwarded-For: 172.17.0.8 ------ Write 346B to [wordpress1] [12b8c6a04031] 172.17.0.7:37557->172.17.0.3:80 (a HTTP/1.1 302 Found Date: Sat, 21 Feb 2015 22:19:18 GMT Trafﬁc between containers

Slide 50

Slide 50 text

Don’t expect existing debugging tools to work Gareth Rushgrove The Bad

Slide 51

Slide 51 text

New tools are emerging. Often with better interfaces Gareth Rushgrove The Good

Slide 52

Slide 52 text

Container Overhead Count the performance penalties

Slide 53

Slide 53 text

Gareth Rushgrove

Slide 54

Slide 54 text

Containers add very little overhead Gareth Rushgrove The Good

Slide 55

Slide 55 text

Gareth Rushgrove

Slide 56

Slide 56 text

Gareth Rushgrove

Slide 57

Slide 57 text

Memory cgroups can be expensive Gareth Rushgrove The Bad

Slide 58

Slide 58 text

By default, the memory subsystem uses 40 bytes of memory per physical page on x86_64 systems. These resources are consumed even if memory is not used in any hierarchy Gareth Rushgrove https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-memory.html

Slide 59

Slide 59 text

Container networking is hard. Overlay networks make it easy. But slow. Gareth Rushgrove The Bad

Slide 60

Slide 60 text

Gareth Rushgrove http://www.generictestdomain.net/docker/weave/networking/stupidity/2015/04/05/weave-is-kinda-slow/

Slide 61

Slide 61 text

Gareth Rushgrove https://arjanschaaf.github.io/is-the-network-the-limit/

Slide 62

Slide 62 text

Gareth Rushgrove The Good

Slide 63

Slide 63 text

Gareth Rushgrove $ qperf between two EC2 c3.8xlarge with 10Gb/s tcp_bw: bw = 1.2 GB/sec udp_lat: latency = 48.1 us $ qperf over weave network using ODP/VXLAN tcp_bw: bw = 1.09 GB/sec udp_lat: latency = 61.9 us

Slide 64

Slide 64 text

Linux containers don’t really contain Gareth Rushgrove The Bad

Slide 65

Slide 65 text

Gareth Rushgrove

Slide 66

Slide 66 text

To get strict isolation guarantees you’re going to wrap them in virtual machines anyway Gareth Rushgrove The Bad

Slide 67

Slide 67 text

Projects like Clear Linux from Intel are innovating in this space Gareth Rushgrove The Good

Slide 68

Slide 68 text

User namespaces (host/container separation) and seccomp (limit syscalls) are coming to Docker Gareth Rushgrove The Good