Scaling, Expending and Extending your app through messaging

This is a talk I gave at the monthly meetup of Ruby Underground in Israel.

I go through the concept of SOA using Ruby/Rails, the reasons to go down this path and the logic behind it.

Transcript

1. Scaling Extending & Expanding Your Application Through Messaging Avi Tzurel Tuesday, October 1, 13
2. ME • Avi Tzurel • avi@kensodev.com • @KensoDev (Twitter, Github, Facebook) Tuesday, October 1, 13
3. What are we talking about? • Current way of building 95% of rails apps • Why is that broken? • Why should you care? • New and improved way of building connected apps Tuesday, October 1, 13
4. Our Use Case Tuesday, October 1, 13
5. Tuesday, October 1, 13
6. Writing a review • Set the review language • Set the review rank (based on some rules) • Update the user score • Was he first to write a full review? (better score) • Update the place score • Update the place score in a travel style • Update the personal place score for user graph Tuesday, October 1, 13
7. The Rails Way • How would you say rails is designed to do this? Tuesday, October 1, 13
8. Yes, I am asking a question What would you say is the rails way? Tuesday, October 1, 13
9. The Rails Way Tuesday, October 1, 13
10. Tuesday, October 1, 13
11. Don’t get smart with me! Rewriting this to use after_commit does not help, this code is shit! Tuesday, October 1, 13
12. Problems Tuesday, October 1, 13
13. Problems • Overloading the Lib Directory • Lib directory gets to be the junk of rails apps Tuesday, October 1, 13
14. It goes on and on.... Tuesday, October 1, 13
15. Problems • Application is bloating beyond your control • Gemfile • Initializers • Models • Super slow startup times • Super slow testing times Tuesday, October 1, 13
16. The slightly better way Tuesday, October 1, 13
17. Tuesday, October 1, 13
18. Again... • Working with workers will make life a bit better • Still lots of bloat in the lib dir • Lots of workers in the application code to load • Where the hell is my logic? • Some engineers put it in the worker, some do classes Tuesday, October 1, 13
19. Tuesday, October 1, 13
20. It Still Sucks! Tuesday, October 1, 13
21. • Going into the better way I will try to be... • Framework Agnostic • Queue system agnostic • I will give you the concept, you make it great for you. • It’s not a one-size-fits-all (things rarely are this way) There’s a better way Tuesday, October 1, 13
22. NOT • Not a single application • Not a single logic location • Not a single test suite • Not a single Gemfile • Not a magic solution for all apps • Not something to start with right away (for a MVP) Tuesday, October 1, 13
23. Tuesday, October 1, 13
24. Tuesday, October 1, 13
25. Publishing Events • Working with Events, not workers or classes • Every model/class can publish any event • There’s no ACK on the other side, it’s just publishing the message to your broker (I said I will be agnostic) • The model/observer responsibility ends here Tuesday, October 1, 13
26. Subscribing • On the other side, your “mini-apps” can subscribe to those events • Events can be unique strings or topics • For example “recommendation_change” is a topic • Any app that cares about this, will register on the event Tuesday, October 1, 13
27. Our Mini apps • LanguageDetectionApp • RecommendationRankingApp • PlaceScoringApp • TravelStyleScoringApp • UserScoringSystem • GraphScoringSystem Tuesday, October 1, 13
28. • Apps can be written in • Full rails apps • Ruby • Sinatra apps • Go • Java more on apps Tuesday, October 1, 13
29. more on apps • Different Gems for different apps • Tests only run on those apps on change • Now, when you change your main app, it runs only those relevant specs • Different API’s, can even be different languages Tuesday, October 1, 13
30. Full Architecture Tuesday, October 1, 13
31. Full Flow • Main app sends recommendation_changed • LanguageDetectionApp registers on this event and checks that the change includes “description” • If change does not include, exits • Change includes, detects the language, sets the column and sends the event again Tuesday, October 1, 13
32. Tuesday, October 1, 13
33. Gotchas • Configuration of apps • Redis connection • DB connection • Mongo Connection • Memcached connection • Deployments Tuesday, October 1, 13
34. • Application config can be stored in... • Chef Data Bags (recommended) • ENV Variables (requires a bit more hassle) we work this way right now and making a move to the previous • Thanks to Avner @ Fiverr for the Chef idea Gotchas Tuesday, October 1, 13
35. Advantages • Full decoupling • Throw complete chunks of code to the trash • No more feature XXX, no problem • Adding another feature dependent on an event, No problem, Main app does not change (doesn’t even have to be redeployed) AMAZING RIGHT? Tuesday, October 1, 13
36. Advantages #2 • Improved response time (any async will give you that though) • Reduced complexity by decoupling • Build apps that use the language best suited instead of one gigantic monster app in Ruby/Rails • Deploy separately, scale separately Tuesday, October 1, 13
37. Read more • Bunny for RabbitMQ • RabbitMQ docs • http://www.rubyinside.com/why-rubyists-should-care- about-messaging-a-high-level-intro-5017.html • http://en.wikipedia.org/wiki/ Fallacies_of_Distributed_Computing • Messaging middleware used at Wikipedia Tuesday, October 1, 13
38. Thank you! Questions? Tuesday, October 1, 13