Containers and Microservices make performance worse

Containers and Microservices make performance worse

Snarky but mainly light-hearted look at how introducing microservices and containers into your infrastructure might just make it less-performant if you don't think about the new challenges.

Presented at the London Web Performance meetup

98234c645fe8c935edc0fec0186d28b8?s=128

Gareth Rushgrove

August 04, 2015
Tweet

Transcript

  1. 18.

    I can optimise the network, the load balancer, the database,

    the application or the client. Easy Gareth Rushgrove
  2. 25.

    Gareth Rushgrove Microservice Microservice Microservice Microservice Microservice Microservice Microservice Microservice

    Network Network Network Network Network Network Network Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network
  3. 27.

    Gareth Rushgrove Microservice Microservice Microservice Microservice Microservice Microservice Microservice Microservice

    Network Network Network Network Network Network Network Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network Load Balancer Network Database Network Network Database
  4. 28.

    In my made-up 8 service architecture we went from 5

    things to optimise up to 32 Gareth Rushgrove The Bad
  5. 29.

    We went from 3 network hops to, er, more depending

    on the request Gareth Rushgrove The Bad
  6. 38.

    Gareth Rushgrove $ free total used free shared buffers cached

    Mem: 1024444 864140 160304 5024 50008 637736 -/+ buffers/cache: 176396 848048 Swap: 473084 16 473068 $ docker exec test-container free total used free shared buffers cached Mem: 1024444 866440 158004 5024 50000 637732 -/+ buffers/cache: 178708 845736 Swap: 473084 16 473068 Can a container use that memory?
  7. 42.

    Gareth Rushgrove $ ps aux USER PID %CPU %MEM VSZ

    RSS TTY STAT START TIME COMMAND ... 999 1807 0.2 11.4 867624 464572 ? Ssl 09:38 0:21 mysqld Is this process in a container?
  8. 43.

    Gareth Rushgrove $ ps -eo ucmd,cgroup COMMAND CGROUP ... mysqld

    9:perf_event:/docker/61e76d2c39121282474ff895b9b3ba2addd775cdea6d2ba89ce76c28 Which container is that?
  9. 46.
  10. 47.

    Gareth Rushgrove $ sudo sysdig -pc -c topprocs_cpu container.name=client CPU%

    Process container.name ---------------------------------------------- 02.69% bash client 31.04% curl client 0.74% sleep client CPU usage in a single container
  11. 48.

    Gareth Rushgrove $ sudo sysdig -pc -c topprocs_net Bytes Process

    Host_pid Container_pid container.name --------------------------------------------------------------- 72.06KB haproxy 7385 13 haproxy 56.96KB docker.io 1775 7039 host 44.45KB mysqld 6995 91 mysql 44.45KB mysqld 6995 99 mysql 29.36KB apache2 7893 124 wordpress1 29.36KB apache2 26895 126 wordpress4 29.36KB apache2 26622 131 wordpress2 29.36KB apache2 27935 132 wordpress3 29.36KB apache2 27306 125 wordpress4 22.23KB mysqld 6995 90 mysqlclient Network bandwidth
  12. 49.

    Gareth Rushgrove $ sudo sysdig -pc -A -c echo_fds "fd.ip=172.17.0.3

    and fd.ip=172.17.0.7" ------ Write 103B to [haproxy] [d468ee81543a] 172.17.0.7:37557->172.17.0.3:80 (hapr GET / HTTP/1.1 User-Agent: curl/7.35.0 Host: 172.17.0.7 Accept: */* X-Forwarded-For: 172.17.0.8 ------ Read 103B from [wordpress1] [12b8c6a04031] 172.17.0.7:37557->172.17.0.3:80 ( GET / HTTP/1.1 User-Agent: curl/7.35.0 Host: 172.17.0.7 Accept: */* X-Forwarded-For: 172.17.0.8 ------ Write 346B to [wordpress1] [12b8c6a04031] 172.17.0.7:37557->172.17.0.3:80 (a HTTP/1.1 302 Found Date: Sat, 21 Feb 2015 22:19:18 GMT Traffic between containers
  13. 58.

    By default, the memory subsystem uses 40 bytes of memory

    per physical page on x86_64 systems. These resources are consumed even if memory is not used in any hierarchy Gareth Rushgrove https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-memory.html
  14. 63.

    Gareth Rushgrove $ qperf between two EC2 c3.8xlarge with 10Gb/s

    tcp_bw: bw = 1.2 GB/sec udp_lat: latency = 48.1 us $ qperf over weave network using ODP/VXLAN tcp_bw: bw = 1.09 GB/sec udp_lat: latency = 61.9 us
  15. 66.

    To get strict isolation guarantees you’re going to wrap them

    in virtual machines anyway Gareth Rushgrove The Bad