Scaling, Expending and Extending your app through messaging

B7d890bed68fa564c18ff00dfd8207cd?s=47 Avi Tzurel
September 30, 2013

Scaling, Expending and Extending your app through messaging

This is a talk I gave at the monthly meetup of Ruby Underground in Israel.

I go through the concept of SOA using Ruby/Rails, the reasons to go down this path and the logic behind it.

B7d890bed68fa564c18ff00dfd8207cd?s=128

Avi Tzurel

September 30, 2013
Tweet

Transcript

  1. 3.

    What are we talking about? • Current way of building

    95% of rails apps • Why is that broken? • Why should you care? • New and improved way of building connected apps Tuesday, October 1, 13
  2. 6.

    Writing a review • Set the review language • Set

    the review rank (based on some rules) • Update the user score • Was he first to write a full review? (better score) • Update the place score • Update the place score in a travel style • Update the personal place score for user graph Tuesday, October 1, 13
  3. 7.

    The Rails Way • How would you say rails is

    designed to do this? Tuesday, October 1, 13
  4. 8.

    Yes, I am asking a question What would you say

    is the rails way? Tuesday, October 1, 13
  5. 11.

    Don’t get smart with me! Rewriting this to use after_commit

    does not help, this code is shit! Tuesday, October 1, 13
  6. 13.

    Problems • Overloading the Lib Directory • Lib directory gets

    to be the junk of rails apps Tuesday, October 1, 13
  7. 15.

    Problems • Application is bloating beyond your control • Gemfile

    • Initializers • Models • Super slow startup times • Super slow testing times Tuesday, October 1, 13
  8. 18.

    Again... • Working with workers will make life a bit

    better • Still lots of bloat in the lib dir • Lots of workers in the application code to load • Where the hell is my logic? • Some engineers put it in the worker, some do classes Tuesday, October 1, 13
  9. 21.

    • Going into the better way I will try to

    be... • Framework Agnostic • Queue system agnostic • I will give you the concept, you make it great for you. • It’s not a one-size-fits-all (things rarely are this way) There’s a better way Tuesday, October 1, 13
  10. 22.

    NOT • Not a single application • Not a single

    logic location • Not a single test suite • Not a single Gemfile • Not a magic solution for all apps • Not something to start with right away (for a MVP) Tuesday, October 1, 13
  11. 25.

    Publishing Events • Working with Events, not workers or classes

    • Every model/class can publish any event • There’s no ACK on the other side, it’s just publishing the message to your broker (I said I will be agnostic) • The model/observer responsibility ends here Tuesday, October 1, 13
  12. 26.

    Subscribing • On the other side, your “mini-apps” can subscribe

    to those events • Events can be unique strings or topics • For example “recommendation_change” is a topic • Any app that cares about this, will register on the event Tuesday, October 1, 13
  13. 27.

    Our Mini apps • LanguageDetectionApp • RecommendationRankingApp • PlaceScoringApp •

    TravelStyleScoringApp • UserScoringSystem • GraphScoringSystem Tuesday, October 1, 13
  14. 28.

    • Apps can be written in • Full rails apps

    • Ruby • Sinatra apps • Go • Java more on apps Tuesday, October 1, 13
  15. 29.

    more on apps • Different Gems for different apps •

    Tests only run on those apps on change • Now, when you change your main app, it runs only those relevant specs • Different API’s, can even be different languages Tuesday, October 1, 13
  16. 31.

    Full Flow • Main app sends recommendation_changed • LanguageDetectionApp registers

    on this event and checks that the change includes “description” • If change does not include, exits • Change includes, detects the language, sets the column and sends the event again Tuesday, October 1, 13
  17. 33.

    Gotchas • Configuration of apps • Redis connection • DB

    connection • Mongo Connection • Memcached connection • Deployments Tuesday, October 1, 13
  18. 34.

    • Application config can be stored in... • Chef Data

    Bags (recommended) • ENV Variables (requires a bit more hassle) we work this way right now and making a move to the previous • Thanks to Avner @ Fiverr for the Chef idea Gotchas Tuesday, October 1, 13
  19. 35.

    Advantages • Full decoupling • Throw complete chunks of code

    to the trash • No more feature XXX, no problem • Adding another feature dependent on an event, No problem, Main app does not change (doesn’t even have to be redeployed) AMAZING RIGHT? Tuesday, October 1, 13
  20. 36.

    Advantages #2 • Improved response time (any async will give

    you that though) • Reduced complexity by decoupling • Build apps that use the language best suited instead of one gigantic monster app in Ruby/Rails • Deploy separately, scale separately Tuesday, October 1, 13
  21. 37.

    Read more • Bunny for RabbitMQ • RabbitMQ docs •

    http://www.rubyinside.com/why-rubyists-should-care- about-messaging-a-high-level-intro-5017.html • http://en.wikipedia.org/wiki/ Fallacies_of_Distributed_Computing • Messaging middleware used at Wikipedia Tuesday, October 1, 13