How to make our concurrency tools suck less exAspArk Evgeny Li

• Multiple parallel requests into the app from 1 user, e.g. to get credentials to each OpenStack computing region • Concurrent HTTP requests from the app, e.g. to combine information from different microservices Challenges

GIL https://twitter.com/reubenbond/status/662061791497744384

• rainbows • batch_api • rack-fiber_pool • eventmachine • sinatra-synchrony • em-synchrony • em-http-request • em-socksify • em-resolv-replace • delayed_job What we currently use Gemfile.lock

Rainbows! is an HTTP server for sleepy Rack applications. It is based on unicorn, but designed to handle applications that expect long request/response times and/or slow clients. For network concurrency, models we currently support are: • EventMachine • FiberPool • ThreadPool • etc. rainbows http://rainbows.bogomips.org/

rainbows http://bogomips.org/rainbows.git/log/

The Batch API is implemented as a Rack middleware. Useful for reducing HTTP overhead by combining requests. sequential – option to execute all operations sequentially, rather than in parallel. This parameter is currently REQUIRED and must be set to true. (In the future the Batch API will offer parallel processing for thread-safe apps, and hence this parameter must be supplied in order to explicitly preserve expected behavior.) batch_api https://github.com/arsduo/batch_api

An event-driven I/O and lightweight concurrency library for Ruby. It provides event-driven I/O using the Reactor pattern, much like Node.js. Pros: • Good performance for networked apps (web server, proxy) • No memory overhead per connection • No threaded programming eventmachine https://github.com/eventmachine/eventmachine

https://www.youtube.com/watch?v=5AxtY4dfuwc RailsConf 2016 - Introduction to Concurrency in Ruby

Cons: • One CPU • Quite a huge lib: 17k + 10k(C++) vs 6k LOC in Celluloid • Hard to debug complex systems: errors, callbacks • Work done per tick should be small • Lack of good and supported libraries around EM eventmachine

Migration trend: • Concurrent-Ruby vs EM, Websocket-Driver vs Faye- Websocket in ActionCable • Bunny vs AMQP • Angelo vs Sinatra-Synchrony eventmachine

How it works: • Loads EventMachine and EM-Synchrony • Inserts the Rack::FiberPool middleware • Adds em-http-request to do concurrent HTTP calls • Patches TCPSocket via EM-Synchrony • Patches Rack::Test to run tests within an EventMachine • Patches Resolv via em-resolv-replace sinatra-synchrony https://github.com/kyledrake/sinatra-synchrony

How it works: • Loads EventMachine and EM-Synchrony • Inserts the Rack::FiberPool middleware • Adds em-http-request to do concurrent HTTP calls • Patches TCPSocket via EM-Synchrony • Patches Rack::Test to run tests within an EventMachine • Patches Resolv via em-resolv-replace sinatra-synchrony https://github.com/kyledrake/sinatra-synchrony

I released sinatra-synchrony so I could use EventMachine without callbacks. Since then, I’ve changed my mind. Ruby developers need to stop using EventMachine. It’s the wrong direction. sinatra-synchrony http://www.slideshare.net/KyleDrake/hybrid-concurrency-patterns

If you are looking for a similar solution, check out Angelo, which is similar to Sinatra but uses Celluloid, Celluloid::IO, Reel, and doesn't compromise real threads or require stupid EM monkey patches. This is a way, way, way better solution than sinatra-synchrony, rack, or EM will ever be. I will not be maintaining this gem anymore. If anyone is interested in maintaining it, feel free to inquire, but I recommend not using EventMachine or sinatra- synchrony anymore. sinatra-synchrony https://github.com/kyledrake/sinatra-synchrony

Collection of convenience classes and primitives to help untangle evented code, plus a number of patched EM clients to make them Fiber aware. https://github.com/igrigorik/em-synchrony/ em-synchrony

!OVOLEHPV\QFKURQ\ DFWLYHUHFRUGUE DPTSUE FRQQHFWLRQBSRROUE FRUHBH[WUE HPKLUHGLVUE HPKWWSUE HPMDFNUE HPPHPFDFKHUE HPPRQJRUE HPPXOWLUE HPUHGLVUE HPUHPFDFKHGUE ƉEHUBLWHUDWRUUE LWHUDWRUUE NH\ERDUGUE PHFKDQL]HUE PRQJRUE PRQJRLGUE P\VTOUE WFSVRFNHWUE WKUHDGUE em-synchrony https://github.com/igrigorik/em-synchrony/tree/master/lib/em-synchrony

Database based asynchronous priority queue system. Pros: • Easy to use • Processes Cons: • No master process and forks • No dashboards • No plugins • Uses Timeout delayed_job https://github.com/collectiveidea/delayed_job

Nobody writes code to defend against an exception being raised on literally any line. That's not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that's unlikely :) Why Ruby’s Timeout is dangerous http://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/

No Timeout in Sidekiq https://github.com/mperham/sidekiq/issues/862#issuecomment-76982399

How do we live with that FODVV2XU'HOD\HG-RE GHIVWDUW LQBHPBV\QFKURQ\^RXUBKDUGBZRUNHUVWDUW` HQG GHILQBHPBV\QFKURQ\ UHWXUQXQOHVVEORFNBJLYHQ" (0V\QFKURQ\GR '-VKRXOGKDQGOHHUURUVEXWVRPHWLPHVLQ(0)LEHU,WHUDWRUWKH\DUHQRWUHVFXHG (0HUURUBKDQGOHU^_HUURU_#HUURU HUURU(0VWRS`VRZHQHHGWRKDQGOHWKHPPDQXDOO\ODWHU \LHOG (0VWRS HQG HQVXUH HQVXUHBWUDQVDFWLRQBƉQLVKHGBFRUUHFWO\ZRUNVZLWK$5 RXUBKDUGBZRUNHUKDQGOHBHUURUBKDUGHU#HUURULI#HUURUKDQGOHHUURUVRXWRI(0 HQG GHIHQVXUHBWUDQVDFWLRQBƉQLVKHGBFRUUHFWO\ WUDQVDFWLRQBPDQDJHU 'HOD\HG-REFRQQHFWLRQWUDQVDFWLRQBPDQDJHU UHWXUQLIWUDQVDFWLRQBPDQDJHUFXUUHQWBWUDQVDFWLRQ WUDQVDFWLRQBPDQDJHUFODVV18//B75$16$&7,21 WUDQVDFWLRQBPDQDJHUUROOEDFNBWUDQVDFWLRQ HQG HQG

• Let’s discuss the problems • Replace EM and all other EMish dependencies • Use Puma to survive under high load • Use Curb / Typhoeus to make concurrent HTTP requests • Contribute to open source ;) Solutions

puma https://github.com/puma/puma

It automatically yields control back to the process when an application thread waits on I/O. If, for example, your application is waiting for an HTTP response from a payments provider, Puma can still accept requests in the Reactor thread or even complete other requests in different application threads. Puma has a "clustered" mode, where it combines its multi- threaded model with Unicorn's multi-process model. In clustered mode, then, Puma can deal with slow requests (thanks to a separate master process whose responsibility it is to download requests and pass them on) and slow application responses (thanks to spawning multiple workers). puma https://www.nateberkopec.com/2015/07/29/scaling-ruby-apps-to-1000-rpm.html

Concurrent HTTP requests https://github.com/exAspArk/concurrent_http_requests

• Less monkey patches and dependencies • Modern supported and reliable libraries • The same performance • Easier to start for new developers Profit

• https://bearmetal.eu/theden/how-do-i-know-whether- my-rails-app-is-thread-safe-or-not/ • https://www.toptal.com/ruby/ruby-concurrency-and- parallelism-a-practical-primer • https://github.com/celluloid • https://github.com/ruby-concurrency/concurrent-ruby • Parallel vs Concurrent • Software Transactional Memory vs locks • Communicating Sequential Processes vs actors • etc. Further reading

Thank you! https://giphy.com/gifs/redbull-26vUsARImRG7qT6ww

Update: Proof of Concept • puma • sinatra • typhoeus + faraday https://gist.github.com/exAspArk/7213b8b12a2340f0b74811d72f9be8ac

Update: Proof of Concept Workers Threads Concurrent requests from server Concurrent requests to server Time 2 2 1 1 1.6 2 2 1 4 1.6 2 2 1 8 4.5 2 2 1 16 6.4 2 2 1 32 13.5 2 2 5 1 1.6 2 2 5 4 1.6 2 2 5 8 4.5 2 2 5 16 6.9 2 2 5 32 14.1

Update: Proof of Concept • Number of concurrent requests from server doesn’t really affect performance • We need to use as many puma workers × threads as number of concurrent requests to server