... but we had to kill Unicorns

Slide 1

Slide 1 text

Unicorn - Wikipedia /ˈjuːnɪkɔːn/ “The unicorn is a legendary animal from European folklore that resembles a white horse with a large, pointed, spiraling horn projecting from its forehead”

Slide 2

Slide 2 text

Unicorn “The unicorn is a legendary animal from European folklore that resembles a white horse with a large, pointed, spiraling horn projecting from its forehead” - Wikipedia /ˈjuːnɪkɔːn/ wrong!

Slide 3

Slide 3 text

Unicorn “Unicorn is a single-threaded HTTP server for Ruby applications designed to only serve fast clients on low- latency, high-bandwidth connections” - Bogomips /ˈjuːnɪkɔːn/

Slide 4

Slide 4 text

Grzegorz Witek Software Developer at SponsorPay @arnvald Simon Kröger VP Engineering at SponsorPay @kroegerberlin ...but we had to kill unicorns

Slide 5

Slide 5 text

we are not hiring! unless you are great 'cos in that case we should talk :)

Slide 6

Slide 6 text

The App integrated inside a game TO BE CHANGED

Slide 7

Slide 7 text

6000m above sea level

Slide 8

Slide 8 text

Old Flow

Slide 9

Slide 9 text

Typical architecture MySQL memcached nginx unicorn rails

Slide 10

Slide 10 text

Product Managers came and started playing with our platform

Slide 11

Slide 11 text

The feature Aggregate offers in real time from different sources TO BE CHANGED

Slide 12

Slide 12 text

Old Flow

Slide 13

Slide 13 text

New Flow

Slide 14

Slide 14 text

Problem 1 concurrency

Slide 15

Slide 15 text

Old Flow

Slide 16

Slide 16 text

libcurl wrappers Typhoeus Curb

Slide 17

Slide 17 text

Problem 2 maths

Slide 18

Slide 18 text

3000 connections/s x 1s = 3000 parallel connections 3000 connections / 10 servers = 300 parallel connections/server

Slide 19

Slide 19 text

300 connections = 300 processes 300 x 0.33GB = 100GB

Slide 20

Slide 20 text

3000 processes = 3000 DB connections 300 processes = 100GB RAM per server

Slide 21

Slide 21 text

and we are growing!

Slide 22

Slide 22 text

MRI + Threads

Slide 23

Slide 23 text

Global VM Lock Global Interpreter Lock No parallelism

Slide 24

Slide 24 text

solutions jRuby

Slide 25

Slide 25 text

solutions jRuby Events

Slide 26

Slide 26 text

solutions jRuby Events New Brave World

Slide 27

Slide 27 text

plain threads

Slide 28

Slide 28 text

MRI + Threads WAT?

Slide 29

Slide 29 text

manually: C API (like mysql2 gem) release VM lock automatically: non-blocking IO

Slide 30

Slide 30 text

Some people, when confronted with a problem, think, "I know, I'll use threads" - and then two they hav erpoblesm.

Slide 31

Slide 31 text

Problem 3 Server

Slide 32

Slide 32 text

Webrick Passenger Yarn Mongrel Thin Unicorn Rainbow ZBattery Puma

Slide 33

Slide 33 text

Webrick Passenger Yarn Mongrel Thin Unicorn Rainbow ZBattery Puma

Slide 34

Slide 34 text

we had to kill the unicorns

Slide 35

Slide 35 text

Webrick Passenger Yarn Mongrel Thin Unicorn Rainbow ZBattery Puma

Slide 36

Slide 36 text

Puma

Slide 37

Slide 37 text

stable version vs development version

Slide 38

Slide 38 text

lack of rolling restarts

Slide 39

Slide 39 text

leaking memory

Slide 40

Slide 40 text

Passenger EE 4.0

Slide 41

Slide 41 text

multi-threaded

Slide 42

Slide 42 text

rolling restarts

Slide 43

Slide 43 text

fewer memory problems

Slide 44

Slide 44 text

live IRB console

Slide 45

Slide 45 text

Problem 4 Threads

Slide 46

Slide 46 text

Ruby is thread-safe Rails is thread-safe

Slide 47

Slide 47 text

what about your code?

Slide 48

Slide 48 text

Thread.current[:var] @@var is the new

Slide 49

Slide 49 text

ulimit -n ~ R_LIMIT_NOFILE

Slide 50

Slide 50 text

200 ulimit -n R_LIMIT_NOFILE

Slide 51

Slide 51 text

300 ulimit -n R_LIMIT_NOFILE

Slide 52

Slide 52 text

400 ulimit -n R_LIMIT_NOFILE

Slide 53

Slide 53 text

more ulimit -n R_LIMIT_NOFILE way >65K

Slide 54

Slide 54 text

limit your DB connections limit your cache server connections

Slide 55

Slide 55 text

pool of connections

Slide 56

Slide 56 text

gem 'connection_pool'

Slide 57

Slide 57 text

Slide 58

Slide 58 text

w e did it!

Slide 59

Slide 59 text

The App now works in multi-threaded mode TO BE CHANGED

Slide 60

Slide 60 text

don’t be afraid of the monster!

Slide 61

Slide 61 text

Thanks

Slide 62

Slide 62 text

Q&A Grzegorz Witek @arnvald Simon Kröger @kroegerberlin