Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling, Extending and Expanding your apps through messaging

Avi Tzurel
January 28, 2014

Scaling, Extending and Expanding your apps through messaging

Avi Tzurel

January 28, 2014
Tweet

More Decks by Avi Tzurel

Other Decks in Technology

Transcript

  1. What are we talking about? • Current way of building

    95% of rails apps • Why is that broken? • Why should you care? • New and improved way of building connected apps
  2. Writing a review • Set the review language • Set

    the review rank (based on some rules) • Update the user score • Was he first to write a full review? (better score) • Update the place score • Update the place score in a travel style • Update the personal place score for user graph
  3. Problems • Application is bloating beyond your control • Gemfile

    • Initializers • Models • Super slow startup times • Super slow testing times
  4. Again... • Working with workers will make life a bit

    better • Still lots of bloat in the lib dir • Lots of workers in the application code to load • Where the hell is my logic? • Some engineers put it in the worker, some do classes
  5. • Going into the better way I will try… •

    Being framework Agnostic • Giving the concept, you make it great for you. • Making sure you know it’s not a one size fits all There’s a better way
  6. NOT • Not a single application • Not a single

    logic location • Not a single test suite • Not a single Gemfile • Not a magic solution for all apps • Not something to start with right away (for a MVP)
  7. Publishing Events • Working with Events, not workers or classes

    • Every model/class can publish any event • There’s no ACK on the other side, it’s just publishing the message to your broker (I said I will be agnostic) • The model/observer responsibility ends here
  8. Subscribing • On the other side, your “mini-apps” can subscribe

    to those events • Events can be unique strings or topics • For example “recommendation_change” is a topic • Any app that cares about this, will register on the event
  9. Our Mini apps • LanguageDetectionApp • RecommendationRankingApp • PlaceScoringApp •

    TravelStyleScoringApp • UserScoringSystem • GraphScoringSystem
  10. • Apps can be written in • Full rails apps

    • Ruby • Sinatra apps • Go • Java more on apps
  11. more on apps • Different Gems for different apps •

    Tests only run on those apps on change • Now, when you change your main app, it runs only those relevant specs • Different API’s, can even be different languages
  12. send.rb #!/usr/bin/env ruby
 # encoding: utf-8
 
 require "bunny"
 


    conn = Bunny.new(:automatically_recover => false)
 conn.start
 
 ch = conn.create_channel
 q = ch.queue("hello")
 
 ch.default_exchange.publish("Hello World!", :routing_key => q.name)
 puts " [x] Sent 'Hello World!'"
 
 conn.close
  13. receive.rb #!/usr/bin/env ruby
 # encoding: utf-8
 
 require "bunny"
 


    conn = Bunny.new(:automatically_recover => false)
 conn.start
 
 ch = conn.create_channel
 q = ch.queue("hello")
 
 begin
 puts " [*] Waiting for messages. To exit press CTRL+C"
 q.subscribe(:block => true) do |delivery_info, properties, body|
 puts " [x] Received #{body}"
 end
 rescue Interrupt => _
 conn.close
 
 exit(0)
 end
  14. new_task.rb #!/usr/bin/env ruby
 # encoding: utf-8
 
 require "bunny"
 


    conn = Bunny.new(:automatically_recover => false)
 conn.start
 
 ch = conn.create_channel
 q = ch.queue("task_queue", :durable => true)
 
 msg = ARGV.empty? ? "Hello World!" : ARGV.join(" ")
 
 q.publish(msg, :persistent => true)
 puts " [x] Sent #{msg}"
 
 conn.close
  15. worker.rb #!/usr/bin/env ruby
 # encoding: utf-8
 
 require "bunny"
 


    conn = Bunny.new(:automatically_recover => false)
 conn.start
 
 ch = conn.create_channel
 q = ch.queue("task_queue", :durable => true)
 
 ch.prefetch(1)
 puts " [*] Waiting for messages. To exit press CTRL+C"
 
 begin
 q.subscribe(:manual_ack => true, :block => true) do |delivery_info, properties, body|
 puts " [x] Received '#{body}'"
 # imitate some work
 sleep body.count(".").to_i
 puts " [x] Done"
 ch.ack(delivery_info.delivery_tag)
 end
 rescue Interrupt => _
 conn.close
 end
  16. Topics • Possibly the most powerful feature of RabbitMQ •

    You can register to a topic via string/partial/regex • Exchange publishes to queues and consumers consume
  17. emit_log_topic.rb #!/usr/bin/env ruby
 # encoding: utf-8
 
 require "bunny"
 


    conn = Bunny.new
 conn.start
 
 ch = conn.create_channel
 x = ch.topic("topic_logs")
 severity = ARGV.shift || "anonymous.info"
 msg = ARGV.empty? ? "Hello World!" : ARGV.join(" ")
 
 x.publish(msg, :routing_key => severity)
 puts " [x] Sent #{severity}:#{msg}"
 
 conn.close
  18. receive_logs_topic.rb #!/usr/bin/env ruby
 # encoding: utf-8
 
 require "bunny"
 


    if ARGV.empty?
 abort "Usage: #{$0} [binding key]"
 end
 
 conn = Bunny.new
 conn.start
 
 ch = conn.create_channel
 x = ch.topic("topic_logs")
 q = ch.queue("", :exclusive => true)
 
 ARGV.each do |severity|
 q.bind(x, :routing_key => severity)
 end
 
 puts " [*] Waiting for logs. To exit press CTRL+C"
 
 begin
 q.subscribe(:block => true) do |delivery_info, properties, body|
 puts " [x] #{delivery_info.routing_key}:#{body}"
 end
 rescue Interrupt => _
 ch.close
 conn.close
 end
  19. Receiving Logs # All Logs
 ruby -rubygems receive_logs_topic.rb "#"
 


    # Logs that start with kern
 ruby -rubygems receive_logs_topic.rb "kern.*"
 
 # Only Critical
 ruby -rubygems receive_logs_topic.rb "*.critical"
 
 # Multiple
 ruby -rubygems receive_logs_topic.rb "kern.*" "*.critical"
 
 # Emiting kern facility log
 ruby -rubygems emit_log_topic.rb "kern.critical" "A critical kernel error"
  20. Gotchas • Configuration of apps • Redis connection • DB

    connection • Mongo Connection • Memcached connection • Deployments
  21. • Application config can be stored in... • Chef Data

    Bags (recommended) • ENV Variables (requires a bit more hassle) we work this way right now and making a move to the previous • Thanks to Avner @ Fiverr for the Chef idea Gotchas
  22. Advantages • Full decoupling • Throw complete chunks of code

    to the trash • No more feature XXX, no problem • Adding another feature dependent on an event, No problem, Main app does not change (doesn’t even have to be redeployed) AMAZING RIGHT?
  23. Advantages #2 • Improved response time (any async will give

    you that though) • Reduced complexity by decoupling • Build apps that use the language best suited instead of one gigantic monster app in Ruby/Rails • Deploy separately, scale separately
  24. Read more • Bunny for RabbitMQ • Sneakers (Awesome!!! Thanks

    to @jundot) • RabbitMQ docs • http://www.rubyinside.com/why-rubyists-should-care- about-messaging-a-high-level-intro-5017.html • http://en.wikipedia.org/wiki/ Fallacies_of_Distributed_Computing • Messaging middleware used at Wikipedia