... but we had to kill Unicorns

Presentation delivered by Grzegorz Witek (@arnvald) and Simon Kröger (@kroegerberlin) on annual european Ruby conference (EuRuKo 2013) in Athens.
The topic of the presentation is a case study, where Sponsorpay developers team switched from Unicorn to multi-threaded server application to serve Ruby on Rails application using MRI 1.9.3.
These beautiful slides were prepared by Juan Vidal (@j2vidal). Thank you, Juan!

Transcript

1. Unicorn
   - Wikipedia
   /ˈjuːnɪkɔːn/
   “The unicorn is a legendary
   animal from European folklore
   that resembles a white horse
   with a large, pointed, spiraling
   horn projecting from its
   forehead”
   

   View Slide

2. Unicorn
   “The unicorn is a legendary
   animal from European folklore
   that resembles a white horse
   with a large, pointed, spiraling
   horn projecting from its
   forehead”
   - Wikipedia
   /ˈjuːnɪkɔːn/
   wrong!
   

   View Slide

3. Unicorn
   “Unicorn is a single-threaded
   HTTP server for Ruby
   applications designed to only
   serve fast clients on low-
   latency, high-bandwidth
   connections”
   - Bogomips
   /ˈjuːnɪkɔːn/
   

   View Slide

4. Grzegorz Witek
   Software Developer at SponsorPay
   @arnvald
   Simon Kröger
   VP Engineering at SponsorPay
   @kroegerberlin
   ...but we had to kill unicorns
   

   View Slide

5. we are not hiring!
   unless you are great 'cos in that case we should talk :)
   

   View Slide

6. The App
   integrated inside a game
   TO BE CHANGED
   

   View Slide

7. 6000m above sea level
   

   View Slide

8. Old Flow
   

   View Slide

9. Typical architecture
   MySQL memcached
   nginx unicorn rails
   

   View Slide

10. Product Managers came and started playing with our platform
    

    View Slide

11. The feature
    Aggregate offers in real time from different sources
    TO BE CHANGED
    

    View Slide

12. Old Flow
    

    View Slide

13. New Flow
    

    View Slide

14. Problem 1
    concurrency
    

    View Slide

15. Old Flow
    

    View Slide

16. libcurl wrappers
    Typhoeus
    Curb
    

    View Slide

17. Problem 2
    maths
    

    View Slide

18. 3000 connections/s x 1s = 3000 parallel connections
    3000 connections / 10 servers = 300 parallel connections/server
    

    View Slide

19. 300 connections = 300 processes
    300 x 0.33GB = 100GB
    

    View Slide

20. 3000 processes = 3000 DB connections
    300 processes = 100GB RAM per server
    

    View Slide

21. and we are growing!
    

    View Slide

22. MRI + Threads
    

    View Slide

23. Global VM Lock
    Global Interpreter Lock
    No parallelism
    

    View Slide

24. solutions
    jRuby
    

    View Slide

25. solutions
    jRuby
    Events
    

    View Slide

26. solutions
    jRuby
    Events
    New Brave World
    

    View Slide

27. plain threads
    

    View Slide

28. MRI + Threads
    WAT?
    

    View Slide

29. manually: C API (like mysql2 gem)
    release VM lock
    automatically: non-blocking IO
    

    View Slide

30. Some people, when confronted with a problem,
    think, "I know, I'll use threads"
    - and then two they hav erpoblesm.
    

    View Slide

31. Problem 3
    Server
    

    View Slide

32. Webrick
    Passenger
    Yarn
    Mongrel
    Thin
    Unicorn
    Rainbow
    ZBattery
    Puma
    

    View Slide

33. Webrick
    Passenger
    Yarn
    Mongrel
    Thin
    Unicorn
    Rainbow
    ZBattery
    Puma
    

    View Slide

34. we had to kill the unicorns
    

    View Slide

35. Webrick
    Passenger
    Yarn
    Mongrel
    Thin
    Unicorn
    Rainbow
    ZBattery
    Puma
    

    View Slide

36. Puma
    

    View Slide

37. stable version vs development version
    

    View Slide

38. lack of rolling restarts
    

    View Slide

39. leaking memory
    

    View Slide

40. Passenger EE
    4.0
    

    View Slide

41. multi-threaded
    

    View Slide

42. rolling restarts
    

    View Slide

43. fewer memory problems
    

    View Slide

44. live IRB console
    

    View Slide

45. Problem 4
    Threads
    

    View Slide

46. Ruby is thread-safe
    Rails is thread-safe
    

    View Slide

47. what about your code?
    

    View Slide

48. Thread.current[:var] @@var
    is the new
    

    View Slide

49. ulimit -n
    ~
    R_LIMIT_NOFILE
    

    View Slide

50. 200 ulimit -n
    R_LIMIT_NOFILE
    

    View Slide

51. 300 ulimit -n
    R_LIMIT_NOFILE
    

    View Slide

52. 400 ulimit -n
    R_LIMIT_NOFILE
    

    View Slide

53. more ulimit -n
    R_LIMIT_NOFILE
    way
    >65K
    

    View Slide

54. limit your DB connections limit your cache server connections
    

    View Slide

55. pool of connections
    

    View Slide

56. gem 'connection_pool'
    

    View Slide

57. more problems?
    

    View Slide

58. w
    e
    did
    it!
    

    View Slide

59. The App
    now works in multi-threaded mode
    TO BE CHANGED
    

    View Slide

60. don’t be afraid of the monster!
    

    View Slide

61. Thanks
    

    View Slide

62. Q&A
    Grzegorz Witek
    @arnvald
    Simon Kröger
    @kroegerberlin
    

    View Slide