Upgrade to Pro — share decks privately, control downloads, hide ads and more …

... but we had to kill Unicorns

... but we had to kill Unicorns

Presentation delivered by Grzegorz Witek (@arnvald) and Simon Kröger (@kroegerberlin) on annual european Ruby conference (EuRuKo 2013) in Athens.
The topic of the presentation is a case study, where Sponsorpay developers team switched from Unicorn to multi-threaded server application to serve Ruby on Rails application using MRI 1.9.3.
These beautiful slides were prepared by Juan Vidal (@j2vidal). Thank you, Juan!

Grzegorz Witek

June 28, 2013
Tweet

More Decks by Grzegorz Witek

Other Decks in Programming

Transcript

  1. Unicorn
    - Wikipedia
    /ˈjuːnɪkɔːn/
    “The unicorn is a legendary
    animal from European folklore
    that resembles a white horse
    with a large, pointed, spiraling
    horn projecting from its
    forehead”

    View full-size slide

  2. Unicorn
    “The unicorn is a legendary
    animal from European folklore
    that resembles a white horse
    with a large, pointed, spiraling
    horn projecting from its
    forehead”
    - Wikipedia
    /ˈjuːnɪkɔːn/
    wrong!

    View full-size slide

  3. Unicorn
    “Unicorn is a single-threaded
    HTTP server for Ruby
    applications designed to only
    serve fast clients on low-
    latency, high-bandwidth
    connections”
    - Bogomips
    /ˈjuːnɪkɔːn/

    View full-size slide

  4. Grzegorz Witek
    Software Developer at SponsorPay
    @arnvald
    Simon Kröger
    VP Engineering at SponsorPay
    @kroegerberlin
    ...but we had to kill unicorns

    View full-size slide

  5. we are not hiring!
    unless you are great 'cos in that case we should talk :)

    View full-size slide

  6. The App
    integrated inside a game
    TO BE CHANGED

    View full-size slide

  7. 6000m above sea level

    View full-size slide

  8. Typical architecture
    MySQL memcached
    nginx unicorn rails

    View full-size slide

  9. Product Managers came and started playing with our platform

    View full-size slide

  10. The feature
    Aggregate offers in real time from different sources
    TO BE CHANGED

    View full-size slide

  11. Problem 1
    concurrency

    View full-size slide

  12. libcurl wrappers
    Typhoeus
    Curb

    View full-size slide

  13. Problem 2
    maths

    View full-size slide

  14. 3000 connections/s x 1s = 3000 parallel connections
    3000 connections / 10 servers = 300 parallel connections/server

    View full-size slide

  15. 300 connections = 300 processes
    300 x 0.33GB = 100GB

    View full-size slide

  16. 3000 processes = 3000 DB connections
    300 processes = 100GB RAM per server

    View full-size slide

  17. and we are growing!

    View full-size slide

  18. MRI + Threads

    View full-size slide

  19. Global VM Lock
    Global Interpreter Lock
    No parallelism

    View full-size slide

  20. solutions
    jRuby

    View full-size slide

  21. solutions
    jRuby
    Events

    View full-size slide

  22. solutions
    jRuby
    Events
    New Brave World

    View full-size slide

  23. plain threads

    View full-size slide

  24. MRI + Threads
    WAT?

    View full-size slide

  25. manually: C API (like mysql2 gem)
    release VM lock
    automatically: non-blocking IO

    View full-size slide

  26. Some people, when confronted with a problem,
    think, "I know, I'll use threads"
    - and then two they hav erpoblesm.

    View full-size slide

  27. Problem 3
    Server

    View full-size slide

  28. Webrick
    Passenger
    Yarn
    Mongrel
    Thin
    Unicorn
    Rainbow
    ZBattery
    Puma

    View full-size slide

  29. Webrick
    Passenger
    Yarn
    Mongrel
    Thin
    Unicorn
    Rainbow
    ZBattery
    Puma

    View full-size slide

  30. we had to kill the unicorns

    View full-size slide

  31. Webrick
    Passenger
    Yarn
    Mongrel
    Thin
    Unicorn
    Rainbow
    ZBattery
    Puma

    View full-size slide

  32. stable version vs development version

    View full-size slide

  33. lack of rolling restarts

    View full-size slide

  34. leaking memory

    View full-size slide

  35. Passenger EE
    4.0

    View full-size slide

  36. multi-threaded

    View full-size slide

  37. rolling restarts

    View full-size slide

  38. fewer memory problems

    View full-size slide

  39. live IRB console

    View full-size slide

  40. Problem 4
    Threads

    View full-size slide

  41. Ruby is thread-safe
    Rails is thread-safe

    View full-size slide

  42. what about your code?

    View full-size slide

  43. Thread.current[:var] @@var
    is the new

    View full-size slide

  44. ulimit -n
    ~
    R_LIMIT_NOFILE

    View full-size slide

  45. 200 ulimit -n
    R_LIMIT_NOFILE

    View full-size slide

  46. 300 ulimit -n
    R_LIMIT_NOFILE

    View full-size slide

  47. 400 ulimit -n
    R_LIMIT_NOFILE

    View full-size slide

  48. more ulimit -n
    R_LIMIT_NOFILE
    way
    >65K

    View full-size slide

  49. limit your DB connections limit your cache server connections

    View full-size slide

  50. pool of connections

    View full-size slide

  51. gem 'connection_pool'

    View full-size slide

  52. more problems?

    View full-size slide

  53. The App
    now works in multi-threaded mode
    TO BE CHANGED

    View full-size slide

  54. don’t be afraid of the monster!

    View full-size slide

  55. Q&A
    Grzegorz Witek
    @arnvald
    Simon Kröger
    @kroegerberlin

    View full-size slide