... but we had to kill Unicorns

Presentation delivered by Grzegorz Witek (@arnvald) and Simon Kröger (@kroegerberlin) on annual european Ruby conference (EuRuKo 2013) in Athens.
The topic of the presentation is a case study, where Sponsorpay developers team switched from Unicorn to multi-threaded server application to serve Ruby on Rails application using MRI 1.9.3.
These beautiful slides were prepared by Juan Vidal (@j2vidal). Thank you, Juan!

Transcript

1. Unicorn - Wikipedia /ˈjuːnɪkɔːn/ “The unicorn is a legendary animal from European folklore that resembles a white horse with a large, pointed, spiraling horn projecting from its forehead”
2. Unicorn “The unicorn is a legendary animal from European folklore that resembles a white horse with a large, pointed, spiraling horn projecting from its forehead” - Wikipedia /ˈjuːnɪkɔːn/ wrong!
3. Unicorn “Unicorn is a single-threaded HTTP server for Ruby applications designed to only serve fast clients on low- latency, high-bandwidth connections” - Bogomips /ˈjuːnɪkɔːn/
4. Grzegorz Witek Software Developer at SponsorPay @arnvald Simon Kröger VP Engineering at SponsorPay @kroegerberlin ...but we had to kill unicorns
5. we are not hiring! unless you are great 'cos in that case we should talk :)
6. The App integrated inside a game TO BE CHANGED
7. 6000m above sea level
8. Old Flow
9. Typical architecture MySQL memcached nginx unicorn rails
10. Product Managers came and started playing with our platform
11. The feature Aggregate offers in real time from different sources TO BE CHANGED
12. Old Flow
13. New Flow
14. Problem 1 concurrency
15. Old Flow
16. libcurl wrappers Typhoeus Curb
17. Problem 2 maths
18. 3000 connections/s x 1s = 3000 parallel connections 3000 connections / 10 servers = 300 parallel connections/server
19. 300 connections = 300 processes 300 x 0.33GB = 100GB
20. 3000 processes = 3000 DB connections 300 processes = 100GB RAM per server
21. and we are growing!
22. MRI + Threads
23. Global VM Lock Global Interpreter Lock No parallelism
24. solutions jRuby
25. solutions jRuby Events
26. solutions jRuby Events New Brave World
27. plain threads
28. MRI + Threads WAT?
29. manually: C API (like mysql2 gem) release VM lock automatically: non-blocking IO
30. Some people, when confronted with a problem, think, "I know, I'll use threads" - and then two they hav erpoblesm.
31. Problem 3 Server
32. Webrick Passenger Yarn Mongrel Thin Unicorn Rainbow ZBattery Puma
33. Webrick Passenger Yarn Mongrel Thin Unicorn Rainbow ZBattery Puma
34. we had to kill the unicorns
35. Webrick Passenger Yarn Mongrel Thin Unicorn Rainbow ZBattery Puma
36. Puma
37. stable version vs development version
38. lack of rolling restarts
39. leaking memory
40. Passenger EE 4.0
41. multi-threaded
42. rolling restarts
43. fewer memory problems
44. live IRB console
45. Problem 4 Threads
46. Ruby is thread-safe Rails is thread-safe
47. what about your code?
48. Thread.current[:var] @@var is the new
49. ulimit -n ~ R_LIMIT_NOFILE
50. 200 ulimit -n R_LIMIT_NOFILE
51. 300 ulimit -n R_LIMIT_NOFILE
52. 400 ulimit -n R_LIMIT_NOFILE
53. more ulimit -n R_LIMIT_NOFILE way >65K
54. limit your DB connections limit your cache server connections
55. pool of connections
56. gem 'connection_pool'
57. more problems?
58. w e did it!
59. The App now works in multi-threaded mode TO BE CHANGED
60. don’t be afraid of the monster!
61. Thanks
62. Q&A Grzegorz Witek @arnvald Simon Kröger @kroegerberlin