questions. - AlthoughPretty nervous, so I might be done in 15... - Ask questions via twitter as they come up - Will have to hand wave over a bunch of stuff due to time constraints, if you want help finding more info about something I mention, ask Monday, April 28, 14
profiler for your rails app. - Used storm - back end data processing - Coordinating writes to cassandra - I will be at the booth tomorrow and thursday - Come talk about some real world storm usage Monday, April 28, 14
serious business. - Took me a while to get into it, I didn’t think that it applied to me. Once I got to know it, it became obviously useful for many applications - I was debating about whether or not going over some use cases up front, but decided against it. - I’m hoping that by first trying to walk through some examples of using it - So, for now let’s just call it a really really powerful worker system Some highlights Monday, April 28, 14
Queue Distributed Queue Internet Storm Worker Storm Worker Storm Worker Zookeeper Zookeeper Storm Nimbus So, this one is kind of a pro and a con, because the operational aspect is not super easy. Monday, April 28, 14
progress in the event of errors. - A lot of systems claim this. Reality is, handling faults is really hard, but I think Storm is one of the few that handles this well, and I will go into some more detail later. Monday, April 28, 14
going to use exponentially weighted moving averages to calculate the rate at which hashtags are being used. - For each hashtag, count the number of occurrences each 5 seconds, then average that number. - Instead of just doing your normal sum all the values and divide by the number of values, we are going to weigh older values exponentially less - Fun fact, linux uses EWMA for calculating the 1m, 5m, and 15min values for CPU load. Monday, April 28, 14
how this might be implemented using Reque or Sidekiq. - To be clear, I’m not putting either of these projects down, we use them. I’m just using them to try to illustrate some problems that storm solves. Monday, April 28, 14
naive and the number of # queries could be reduced. def perform(tweet) tags = extract_hashtags(tweet.body) tags.each do |hashtag| existing = HashTag.find_or_new_by_name(hashtag) existing.update_ewma(Time.now) existing.save! end end end Monday, April 28, 14
naive and the number of # queries could be reduced. def perform(tweet) tags = extract_hashtags(tweet.body) tags.each do |hashtag| existing = HashTag.find_or_new_by_name(hashtag) existing.update_ewma(Time.now) existing.save! end end end Monday, April 28, 14
naive and the number of # queries could be reduced. def perform(tweet) tags = extract_hashtags(tweet.body) tags.each do |hashtag| existing = HashTag.find_or_new_by_name(hashtag) existing.update_ewma(Time.now) existing.save! end end end Monday, April 28, 14
catchup(now) tick until time >= now.to_i end def tick interval = 5 # in seconds # Compute the rate this interval (aka the num # of occurences this tick) instant_rate = uncounted / interval # Reset the count self.uncounted = 0 self.rate += ALPHA * (instant_rate - self.rate) self.time += interval end Monday, April 28, 14
problem. Our EWMA algorithm requires us to update the rate value of the hashtag every 5 seconds. - This works as long as there are tweets that arrive containing the hashtag. However, what if that isn’t the case? We need to run another job to ensure that the hashtags keep getting their rate values updated even when no tweets arrive. Monday, April 28, 14
< ?", now - CUTOFF) tags = HashTag.where("time < ?", now).all tags.each do |hashtag| hashtag.catchup(now) hashtag.save! end end end - Cool, this should conceptually work. - Though, I haven’t actually ran any of this code. - There is one more super important question to ask Monday, April 28, 14
Alright, now we’re talking. Got 3 workers going, are we ready to handle twitter’s 50k+ ps tweet firehose? - Well.... maybe not quite. But no worries, we got more tricks up our sleeves. - Let’s add some caching. - We’re going to cache the hashtag records in memory in each worker. - Everybody knows caching is easy... Monday, April 28, 14
tags = extract_hashtags(tweet.body) tags.each do |hashtag| unless existing = @hashtags[hashtag] @hashtags[hashtag] = HashTag.new_by_name(hashtag) existing = @hashtags[hashtag] end existing.update_ewma(Time.now) existing.save! end end end Monday, April 28, 14
2 count 2 count 2 @tomdale Just landed! #railsconf? count 2 count 2 - Can’t cache hashtags - This has to do with how these systems work. Workers pop the next available message from the queue and process it. - Workers are assumed to bootstrap their state each time. - We could probably reduce each worker to effectively run a single (large) SQL query, but that would still require a SQL query for each tweet. - Punting coordination to the database, and that’s where the bottleneck will end up. Even though we have many parallel workers, the database can only process one update at a time. - There are still many things we can do to fix this up. Monday, April 28, 14
2 count 2 count 2 @tomdale Just landed! #railsconf? count 2 count 2 - Can’t cache hashtags - This has to do with how these systems work. Workers pop the next available message from the queue and process it. - Workers are assumed to bootstrap their state each time. - We could probably reduce each worker to effectively run a single (large) SQL query, but that would still require a SQL query for each tweet. - Punting coordination to the database, and that’s where the bottleneck will end up. Even though we have many parallel workers, the database can only process one update at a time. - There are still many things we can do to fix this up. Monday, April 28, 14
- That is what storm is trying to do - Make these sorts of problems easier. - Let’s start diving in. - going to start by going over some more abstract concepts, but hopefully I’ll be able to tie it together with examples Monday, April 28, 14
of tubes through which data gets piped, but storm calls the pipes streams and the data tuples. - A tuple is just a list of values. The values can be anything you want. Strings, integers, or objects of any complexity. The only limitation is that you can serialize them. You can define custom serializers for any type of object. I’m not going to get too much into the specifics of serialization though. - The bulk of storm is just a set of primitives to transform the streams of data. Monday, April 28, 14
They are the entry point into storm. Anything that reads from the outside world - Read from queues (Redis, SQS, etc..) - Read directly from the twitter API - Read from databases - HTTP Get requests - Time of day State objects are the opposite. They are the stream “endpoints” They allow the results of the data transformations to be available outside of storm. - Anything that “writes” outside of storm - Writes to the DB - HTTP POST requests - Pushing to external queues - Sending email Monday, April 28, 14
haven’t done anything interesting yet. - There is no point really to just read the data in one end and write it as is out the other end. Spout State Stream Monday, April 28, 14
we are at. - We have a spout that feeds data in - We can run it through some transforms - The data flows through and ends up at a state, where it will exit storm somehow, usually by being written to a database. Monday, April 28, 14
Filter Join Join State - Add annotations for filter (1 tweet per user) - Aggregate by hashtag - We’ll look at how to write these and how to hook them all together Monday, April 28, 14
transforms is called a topology - Represents the execution - I’m not going to talk much about deployment, but basically, you define this topology Monday, April 28, 14
provided via OSS packages - Redis spout, SQS spout, Kestrel, Kafka, etc... - There already are libraries of provided transformations - Transformations can be made generic, packaged up, reused, and shared - Instead of listing all the available spouts, I’m going to show how to implement them Monday, April 28, 14
f("msg")). each(f("msg"), MyLogger.new, f()) end def f(*names) Fields.new(*names) end class MyQueueMsgDeserializer < BaseFunction def execute(tuple, output) bytes = tuple.get_value_by_field("bytes") msg = Msg.new(JSON.parse(bytes)) output.emit(Values.new(msg)) end end Monday, April 28, 14
results - Again, I’m going to jump into the low level implementation of a state. There are higher level ones that can automatically persist to memcached or cassandra, or riak, or anything Monday, April 28, 14
def persist_awesomely(msg) awesome_msg = MyAwesomeMsg.new(msg) awesome_msg.save! end end Only begin_commit / commit are required by the state Monday, April 28, 14
input.get_value_by_field("msg") my_basic_state.persist_awesomely(msg) end end - Define a state updater. - This is what receives the tuples off the stream and writes them to the state. Monday, April 28, 14
summary end def aggregate(summary, tuple, output) hashtag = tuple.get_value_by_field("hashtag") summary[hashtag] ||= 0 hashtag += 1 end def complete(summary, output) summary.each do |hashtag, count| output.emit(Values.new(hashtag, count)) end end end Monday, April 28, 14
summary end def aggregate(summary, tuple, output) hashtag = tuple.get_value_by_field("hashtag") summary[hashtag] ||= 0 hashtag += 1 end def complete(summary, output) summary.each do |hashtag, count| output.emit(Values.new(hashtag, count)) end end end When does this run? Streams are an unbounded sequence of tuples, so when is it “complete”? Monday, April 28, 14
tuple, could be 1MM tuples. More is generally better. The spout will get fetch a number of messages to make the batch and send it downstream. Aggregation completion happens at the end of the batch (after the aggregation transform has seen all tuples in the batch) State begin_commit / commit Monday, April 28, 14
happen It’s important to note that it is OK to send all tuples w/ the same hashtag to the same state partition. The state partition will not be overloaded because we first aggregated. Worst case scenario, the state will receive N tuples per hashtag where N is the number of partitions Monday, April 28, 14
to happen It’s important to note that it is OK to send all tuples w/ the same hashtag to the same state partition. The state partition will not be overloaded because we first aggregated. Worst case scenario, the state will receive N tuples per hashtag where N is the number of partitions Monday, April 28, 14
to happen It’s important to note that it is OK to send all tuples w/ the same hashtag to the same state partition. The state partition will not be overloaded because we first aggregated. Worst case scenario, the state will receive N tuples per hashtag where N is the number of partitions Monday, April 28, 14
to happen It’s important to note that it is OK to send all tuples w/ the same hashtag to the same state partition. The state partition will not be overloaded because we first aggregated. Worst case scenario, the state will receive N tuples per hashtag where N is the number of partitions Monday, April 28, 14
want to happen It’s important to note that it is OK to send all tuples w/ the same hashtag to the same state partition. The state partition will not be overloaded because we first aggregated. Worst case scenario, the state will receive N tuples per hashtag where N is the number of partitions Monday, April 28, 14
want to happen It’s important to note that it is OK to send all tuples w/ the same hashtag to the same state partition. The state partition will not be overloaded because we first aggregated. Worst case scenario, the state will receive N tuples per hashtag where N is the number of partitions Monday, April 28, 14
want to happen It’s important to note that it is OK to send all tuples w/ the same hashtag to the same state partition. The state partition will not be overloaded because we first aggregated. Worst case scenario, the state will receive N tuples per hashtag where N is the number of partitions Monday, April 28, 14
we want to happen It’s important to note that it is OK to send all tuples w/ the same hashtag to the same state partition. The state partition will not be overloaded because we first aggregated. Worst case scenario, the state will receive N tuples per hashtag where N is the number of partitions Monday, April 28, 14
we want to happen It’s important to note that it is OK to send all tuples w/ the same hashtag to the same state partition. The state partition will not be overloaded because we first aggregated. Worst case scenario, the state will receive N tuples per hashtag where N is the number of partitions Monday, April 28, 14
we want to happen It’s important to note that it is OK to send all tuples w/ the same hashtag to the same state partition. The state partition will not be overloaded because we first aggregated. Worst case scenario, the state will receive N tuples per hashtag where N is the number of partitions Monday, April 28, 14
TrendingTopicFactory.new, f("hashtag", "count"), TrendingTopicUpdater.new) partition ensures that all tuples with the same hashtag get persisted on the same server Monday, April 28, 14
your use case. Maybe 8 is appropriate, maybe 512 is. Knowing how a topology gets executed on the cluster might help Let’s talk about that real quick Monday, April 28, 14
Thread Thread P P P P P P P P P P P P P P P P P P P P P P P P P P P Arbitrary number, it’s configurable, so tweak for your use case. Maybe 8 is appropriate, maybe 512 is. Monday, April 28, 14
Thread Thread P P P P P P P P P P P P P P P P P P P P P P P P P P P Arbitrary number, it’s configurable, so tweak for your use case. Maybe 8 is appropriate, maybe 512 is. Monday, April 28, 14
Maybe 8 is appropriate, maybe 512 is. Server Server Thread Thread Thread Thread Thread Thread P P P P P P P P P P P P P P P P P P P P P P P P P P P Monday, April 28, 14
Thread Thread P P P P P P P P P P P P P P P P P P P P P P P P P P P Arbitrary number, it’s configurable, so tweak for your use case. Maybe 8 is appropriate, maybe 512 is. New Server Monday, April 28, 14
Thread Thread P P P P P P P P P P P P P P P P P P P P P Server Thread Thread Thread P P P P P P (unlike my slide, which looks pretty off balanced) STORM REBALANCES Monday, April 28, 14
- The question is how do we handle it - Will we recover? - Will our system end up inconsistent? - Will we lose availability? - Handling failure is probably the hardest part of building a robust distributed system. - We have all built distributed systems. - If you have built a rails app, then you have built a distributed system. - The browser talks to the server which talks to the database. - Failure can happen at any stage - Consider a signup form, what happens if the user hits submit and the request failed? - Did the request reach the rails app? - Did the rails app start processing it? - At what point did the request fail? - Was it before writing to the DB? - Was it after? - If the user attempts to signup again, what will happen? - Will the user be successful? - Will the user get an error stating that there already is an account with the given email address? - How can we, as developers, prevent this from happening? - I bring up such a “simple” case, because it just gets more complex from here. Monday, April 28, 14
Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple - This is more what tends to happen - You get one input tuple, and throughout the processing, it spawns off more tuples - There isn’t really any upper bound to how many tuples can be fanned out from an original source Monday, April 28, 14
Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple - So what do you do in this case? - Do you track the status of every single message? The book keeping will be huge. - Things get even messier when you do joins. - AKA, when a tuple has more than one parent. - What happens then? You can’t back out of the processing Monday, April 28, 14
naive and the number of # queries could be reduced. def perform(tweet) user = User.find_by_username(tweet.user) tags = user.new_hashtags_mentioned_this_hour(tweet.body) tags do |hashtag| existing = HashTag.find_or_new_by_name(hashtag) existing.update_ewma(Time.now) existing.save! end end end BOOM Monday, April 28, 14
||= load_hashtags_for(partition_ids) end def commit(transaction_id) tick_all_tags(transaction_id) delete_stale_hashtags end def update(name, count, timestamp) hashtag = (@hashtags[name] ||= HashTag.new) return if hashtag.last_txid == @txid hashtag.update_ewma(count, timestamp) end end Monday, April 28, 14
||= load_hashtags_for(partition_ids) end def commit(transaction_id) tick_all_tags(transaction_id) delete_stale_hashtags end def update(name, count, timestamp) hashtag = (@hashtags[name] ||= HashTag.new) return if hashtag.last_txid == @txid hashtag.update_ewma(count, timestamp) end end Monday, April 28, 14
||= load_hashtags_for(partition_ids) end def commit(transaction_id) tick_all_tags(transaction_id) delete_stale_hashtags end def update(name, count, timestamp) hashtag = (@hashtags[name] ||= HashTag.new) return if hashtag.last_txid == @txid hashtag.update_ewma(count, timestamp) end end Monday, April 28, 14
||= load_hashtags_for(partition_ids) end def commit(transaction_id) tick_all_tags(transaction_id) delete_stale_hashtags end def update(name, count, timestamp) hashtag = (@hashtags[name] ||= HashTag.new) return if hashtag.last_txid == @txid hashtag.update_ewma(count, timestamp) end end Monday, April 28, 14
use Time.now in your transforms. - There are a few options - You can make a spout that’s entire job is to get Time.now for a batch ID and save it somewhere so that if the batch is re- emmitted, it reemits the same value Another option: Batch time dependent on input. Each tweet probably has a time associated with it, so write some function of all input tweets to compute a value for “now” Monday, April 28, 14
than traditional queue - Send messages to it, it appends them to a queue - The application has a cursor, starts at zero, and the application asks for messages from Kafka at that cursor. - It’s the application’s job to manage the cursor - One queue reader? Since no coordination - Kafka queues are very partitioned. Each reader reads a different partition. - Fits in well with storm since storm spouts are partitioned. Each storm partition reads from it’s own kafka partition - Nice properties: Replay Monday, April 28, 14
yuzneri
yug1224
syumai
freee
hirokiyamamoto14
dachi023
hideg
mazrean
l2hyunwoo
andpad
izumin5210
yimajo
mayatellez
davetheseo
jlugia
addyosmani
.png){kind=avatar}
helenjbeal
lara
apolaine
eileencodes
greggifford
takai
destraynor
bryan
