Tweet
Tweet

Slide 1

Slide 1 text

Scaling Extending & Expanding Your Application Through Messaging Avi Tzurel Tuesday, October 1, 13

Slide 2

Slide 2 text

ME • Avi Tzurel • [email protected] • @KensoDev (Twitter, Github, Facebook) Tuesday, October 1, 13

Slide 3

Slide 3 text

What are we talking about? • Current way of building 95% of rails apps • Why is that broken? • Why should you care? • New and improved way of building connected apps Tuesday, October 1, 13

Slide 4

Slide 4 text

Our Use Case Tuesday, October 1, 13

Slide 5

Slide 5 text

Tuesday, October 1, 13

Slide 6

Slide 6 text

Writing a review • Set the review language • Set the review rank (based on some rules) • Update the user score • Was he first to write a full review? (better score) • Update the place score • Update the place score in a travel style • Update the personal place score for user graph Tuesday, October 1, 13

Slide 7

Slide 7 text

The Rails Way • How would you say rails is designed to do this? Tuesday, October 1, 13

Slide 8

Slide 8 text

Yes, I am asking a question What would you say is the rails way? Tuesday, October 1, 13

Slide 9

Slide 9 text

The Rails Way Tuesday, October 1, 13

Slide 10

Slide 10 text

Tuesday, October 1, 13

Slide 11

Slide 11 text

Don’t get smart with me! Rewriting this to use after_commit does not help, this code is shit! Tuesday, October 1, 13

Slide 12

Slide 12 text

Problems Tuesday, October 1, 13

Slide 13

Slide 13 text

Problems • Overloading the Lib Directory • Lib directory gets to be the junk of rails apps Tuesday, October 1, 13

Slide 14

Slide 14 text

It goes on and on.... Tuesday, October 1, 13

Slide 15

Slide 15 text

Problems • Application is bloating beyond your control • Gemfile • Initializers • Models • Super slow startup times • Super slow testing times Tuesday, October 1, 13

Slide 16

Slide 16 text

The slightly better way Tuesday, October 1, 13

Slide 17

Slide 17 text

Tuesday, October 1, 13

Slide 18

Slide 18 text

Again... • Working with workers will make life a bit better • Still lots of bloat in the lib dir • Lots of workers in the application code to load • Where the hell is my logic? • Some engineers put it in the worker, some do classes Tuesday, October 1, 13

Slide 19

Slide 19 text

Tuesday, October 1, 13

Slide 20

Slide 20 text

It Still Sucks! Tuesday, October 1, 13

Slide 21

Slide 21 text

• Going into the better way I will try to be... • Framework Agnostic • Queue system agnostic • I will give you the concept, you make it great for you. • It’s not a one-size-fits-all (things rarely are this way) There’s a better way Tuesday, October 1, 13

Slide 22

Slide 22 text

NOT • Not a single application • Not a single logic location • Not a single test suite • Not a single Gemfile • Not a magic solution for all apps • Not something to start with right away (for a MVP) Tuesday, October 1, 13

Slide 23

Slide 23 text

Tuesday, October 1, 13

Slide 24

Slide 24 text

Tuesday, October 1, 13

Slide 25

Slide 25 text

Publishing Events • Working with Events, not workers or classes • Every model/class can publish any event • There’s no ACK on the other side, it’s just publishing the message to your broker (I said I will be agnostic) • The model/observer responsibility ends here Tuesday, October 1, 13

Slide 26

Slide 26 text

Subscribing • On the other side, your “mini-apps” can subscribe to those events • Events can be unique strings or topics • For example “recommendation_change” is a topic • Any app that cares about this, will register on the event Tuesday, October 1, 13

Slide 27

Slide 27 text

Our Mini apps • LanguageDetectionApp • RecommendationRankingApp • PlaceScoringApp • TravelStyleScoringApp • UserScoringSystem • GraphScoringSystem Tuesday, October 1, 13

Slide 28

Slide 28 text

• Apps can be written in • Full rails apps • Ruby • Sinatra apps • Go • Java more on apps Tuesday, October 1, 13

Slide 29

Slide 29 text

more on apps • Different Gems for different apps • Tests only run on those apps on change • Now, when you change your main app, it runs only those relevant specs • Different API’s, can even be different languages Tuesday, October 1, 13

Slide 30

Slide 30 text

Full Architecture Tuesday, October 1, 13

Slide 31

Slide 31 text

Full Flow • Main app sends recommendation_changed • LanguageDetectionApp registers on this event and checks that the change includes “description” • If change does not include, exits • Change includes, detects the language, sets the column and sends the event again Tuesday, October 1, 13

Slide 32

Slide 32 text

Tuesday, October 1, 13

Slide 33

Slide 33 text

Gotchas • Configuration of apps • Redis connection • DB connection • Mongo Connection • Memcached connection • Deployments Tuesday, October 1, 13

Slide 34

Slide 34 text

• Application config can be stored in... • Chef Data Bags (recommended) • ENV Variables (requires a bit more hassle) we work this way right now and making a move to the previous • Thanks to Avner @ Fiverr for the Chef idea Gotchas Tuesday, October 1, 13

Slide 35

Slide 35 text

Advantages • Full decoupling • Throw complete chunks of code to the trash • No more feature XXX, no problem • Adding another feature dependent on an event, No problem, Main app does not change (doesn’t even have to be redeployed) AMAZING RIGHT? Tuesday, October 1, 13

Slide 36

Slide 36 text

Advantages #2 • Improved response time (any async will give you that though) • Reduced complexity by decoupling • Build apps that use the language best suited instead of one gigantic monster app in Ruby/Rails • Deploy separately, scale separately Tuesday, October 1, 13

Slide 37

Slide 37 text

Read more • Bunny for RabbitMQ • RabbitMQ docs • http://www.rubyinside.com/why-rubyists-should-care- about-messaging-a-high-level-intro-5017.html • http://en.wikipedia.org/wiki/ Fallacies_of_Distributed_Computing • Messaging middleware used at Wikipedia Tuesday, October 1, 13

Slide 38

Slide 38 text

Thank you! Questions? Tuesday, October 1, 13