Upgrade to Pro — share decks privately, control downloads, hide ads and more …

... but we had to kill Unicorns

... but we had to kill Unicorns

Presentation delivered by Grzegorz Witek (@arnvald) and Simon Kröger (@kroegerberlin) on annual european Ruby conference (EuRuKo 2013) in Athens.
The topic of the presentation is a case study, where Sponsorpay developers team switched from Unicorn to multi-threaded server application to serve Ruby on Rails application using MRI 1.9.3.
These beautiful slides were prepared by Juan Vidal (@j2vidal). Thank you, Juan!

Afa6e65b91b46f3317c3d3e94e9255f2?s=128

Grzegorz Witek

June 28, 2013
Tweet

Transcript

  1. Unicorn - Wikipedia /ˈjuːnɪkɔːn/ “The unicorn is a legendary animal

    from European folklore that resembles a white horse with a large, pointed, spiraling horn projecting from its forehead”
  2. Unicorn “The unicorn is a legendary animal from European folklore

    that resembles a white horse with a large, pointed, spiraling horn projecting from its forehead” - Wikipedia /ˈjuːnɪkɔːn/ wrong!
  3. Unicorn “Unicorn is a single-threaded HTTP server for Ruby applications

    designed to only serve fast clients on low- latency, high-bandwidth connections” - Bogomips /ˈjuːnɪkɔːn/
  4. Grzegorz Witek Software Developer at SponsorPay @arnvald Simon Kröger VP

    Engineering at SponsorPay @kroegerberlin ...but we had to kill unicorns
  5. we are not hiring! unless you are great 'cos in

    that case we should talk :)
  6. The App integrated inside a game TO BE CHANGED

  7. 6000m above sea level

  8. Old Flow

  9. Typical architecture MySQL memcached nginx unicorn rails

  10. Product Managers came and started playing with our platform

  11. The feature Aggregate offers in real time from different sources

    TO BE CHANGED
  12. Old Flow

  13. New Flow

  14. Problem 1 concurrency

  15. Old Flow

  16. libcurl wrappers Typhoeus Curb

  17. Problem 2 maths

  18. 3000 connections/s x 1s = 3000 parallel connections 3000 connections

    / 10 servers = 300 parallel connections/server
  19. 300 connections = 300 processes 300 x 0.33GB = 100GB

  20. 3000 processes = 3000 DB connections 300 processes = 100GB

    RAM per server
  21. and we are growing!

  22. MRI + Threads

  23. Global VM Lock Global Interpreter Lock No parallelism

  24. solutions jRuby

  25. solutions jRuby Events

  26. solutions jRuby Events New Brave World

  27. plain threads

  28. MRI + Threads WAT?

  29. manually: C API (like mysql2 gem) release VM lock automatically:

    non-blocking IO
  30. Some people, when confronted with a problem, think, "I know,

    I'll use threads" - and then two they hav erpoblesm.
  31. Problem 3 Server

  32. Webrick Passenger Yarn Mongrel Thin Unicorn Rainbow ZBattery Puma

  33. Webrick Passenger Yarn Mongrel Thin Unicorn Rainbow ZBattery Puma

  34. we had to kill the unicorns

  35. Webrick Passenger Yarn Mongrel Thin Unicorn Rainbow ZBattery Puma

  36. Puma

  37. stable version vs development version

  38. lack of rolling restarts

  39. leaking memory

  40. Passenger EE 4.0

  41. multi-threaded

  42. rolling restarts

  43. fewer memory problems

  44. live IRB console

  45. Problem 4 Threads

  46. Ruby is thread-safe Rails is thread-safe

  47. what about your code?

  48. Thread.current[:var] @@var is the new

  49. ulimit -n ~ R_LIMIT_NOFILE

  50. 200 ulimit -n R_LIMIT_NOFILE

  51. 300 ulimit -n R_LIMIT_NOFILE

  52. 400 ulimit -n R_LIMIT_NOFILE

  53. more ulimit -n R_LIMIT_NOFILE way >65K

  54. limit your DB connections limit your cache server connections

  55. pool of connections

  56. gem 'connection_pool'

  57. more problems?

  58. w e did it!

  59. The App now works in multi-threaded mode TO BE CHANGED

  60. don’t be afraid of the monster!

  61. Thanks

  62. Q&A Grzegorz Witek @arnvald Simon Kröger @kroegerberlin