Boston Storm Users: Summingbird, Scala and Storm

@summingbird
Friday, September 27, 13

View Slide


View Slide

Oscar Boykin - @posco
Sam Ritchie - @sritchie
Ashu Singhal - @daashu

View Slide

- What is Summingbird?
- What can it do today?
- Why you should use it to write Storm!
- Currently deployed systems
- Upcoming Features

View Slide

Vision

View Slide

- 200M+ Active Monthly Users
- 500M Tweets / Day
- Several 1K+ node Hadoop clusters
- THE Realtime Company (ostensibly)
Twitter’s Scale

View Slide

Write your logic once.

View Slide

Solve systems problems once.

View Slide

Make non-trivial
realtime compute
as accessible
as Scalding.

View Slide

- Declarative Streaming Map/Reduce DSL
- Realtime platform that runs on Storm.
- Batch platform that runs on Hadoop.
- Batch / Realtime Hybrid platform
What is Summingbird?

View Slide

Why does Storm Care?

View Slide

public class WordCountTopology {
public static class SplitSentence extends ShellBolt implements IRichBolt {
public SplitSentence() {
super("python", "splitsentence.py");
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word"));
}
@Override
public Map getComponentConfiguration() {
return null;
}
}
public static class WordCount extends BaseBasicBolt {
Map counts = new HashMap();
@Override
public void execute(Tuple tuple, BasicOutputCollector collector) {
String word = tuple.getString(0);
Integer count = counts.get(word);
if(count==null) count = 0;
count++;
counts.put(word, count);
collector.emit(new Values(word, count));
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word", "count"));
}
}
public static void main(String[] args) throws Exception {
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new RandomSentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8)
.shuffleGrouping("spout");
builder.setBolt("count", new WordCount(), 12)
.fieldsGrouping("split", new Fields("word"));
Config conf = new Config();
conf.setDebug(true);
if(args!=null && args.length > 0) {
conf.setNumWorkers(3);
StormSubmitter.submitTopology(args[0], conf, builder.createTopology());
} else {
conf.setMaxTaskParallelism(3);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("word-count", conf, builder.createTopology());
Thread.sleep(10000);
cluster.shutdown();
}
}
}

View Slide

def tokenize(text: String) :
TraversableOnce[String] =
text.toLowerCase
.replaceAll("[^a-zA-Z0-9\\s]", "")
.split("\\s+")
def wordCount[P <: Platform[P]](
source: Producer[P, Status],
store: P#Store[String, Long]) =
source
.filter(_.getText != null)
.flatMap { tweet: Status =>
tokenize(tweet.getText).map(_ -> 1L)
}.sumByKey(store)
}

View Slide

Map/Reduce
f1 f1 f2 f2 f2
+ + + + +
Event Stream 1 Event Stream 2
FlatMappers
Reducers
Storage (Memcache / ElephantDB)

View Slide

Functions, not Bolts!
flatMap: T => TraversableOnce[U]
// g: (x: T => U)
map(x) = flatMap(x => List(g(x))
// pred: T => Boolean
filter(x) = flatMap { x =>
if (pred(x)) List(x) else Nil
}

View Slide

- Source[+T]
- Store[-K, V]
- Sink[-T]
- Service[-K, +V]

View Slide

- Source[+T]
- Store[-K, V]
- Sink[-T]
- Service[-K, +V]
The Four Ss!

View Slide

Source[+T] = Spout[(Long, T)]
Store[-K, V] = StormStore[K, V]
Sink[-T] = () => (T => Future[Unit])
Service[-K, +V] = StormService[K, V]
Plan[T] = StormTopology
The Storm Platform

View Slide


View Slide

Type Safety

View Slide


View Slide

trait Spout[+T] {
def getSpout: IRichSpout
def flatMap[U](fn: T => TraversableOnce[U]): Spout[U]
}

View Slide

val spout: Spout[String] =
Spout.fromTraversable {
List("call" "me" "ishmael")
}
val evenCountSpout: Spout[Int] =
spout.map(_.size).filter(_ % 2 == 0)
val characterSpout: Spout[String] =
spout.flatMap { s =>
s.toSeq.map(_.toString)
}

View Slide

val characterSpout: Spout[String] =
spout.flatMap { s =>
s.toSeq.map(_.toString)
}
val builder = new TopologyBuilder
builder.setSpout("1", characterSpout.getSpout, 1)
builder.setBolt("2", new TestGlobalCount())
.globalGrouping("1")
val topo: StormTopology = builder.createTopology

View Slide


View Slide

trait ReadableStore[-K, +V] extends Closeable {
def get(k: K): Future[Option[V]]
def multiGet[K1 <: K](ks: Set[K1]): Map[K1, Future[Option[V]]]
override def close { }
}
trait Store[-K, V] extends ReadableStore[K, V] {
def put(kv: (K, Option[V])): Future[Unit]
def multiPut[K1 <: K](kvs: Map[K1, Option[V]])
: Map[K1, Future[Unit]]
}

View Slide

trait MergeableStore[-K, V] extends Store[K, V] {
def monoid: Monoid[V]
def merge(kv: (K, V)): Future[Unit]
def multiMerge[K1 <: K](kvs: Map[K1, V])
: Map[K1, Future[Unit]]
}

View Slide

What values are allowed?

View Slide

trait Monoid[V] {
def zero: V
def plus(l: V, r: V): V
}

View Slide

• Tons O’Monoids:
• CMS,
HyperLogLog,
ExponentialMA,
BloomFilter,
Moments,
MinHash, TopK

View Slide


View Slide

Associativity

View Slide

;; 7 steps
a0 + a1 + a2 + a3 + a4 + a5 + a6 + a7

View Slide

;; 7 steps
(+ a0 a1 a2 a3 a4 a5 a6 a7)

View Slide

;; 5 steps
(+ (+ a0 a1)
(+ a2 a3)
(+ a4 a5)
(+ a6 a7))

View Slide

;; 3 steps
(+ (+ (+ a0 a1)
(+ a2 a3))
(+ (+ a4 a5)
(+ a6 a7)))

View Slide

Parallelism
Associativity

View Slide

Clients
Storehaus

View Slide

Tweet Embed Counts

View Slide


View Slide

f f f
+ + + + +
Tweets
(Flat)Mappers
Reducers
HDFS/Queue
HDFS/Queue
reduce: (x,y) =>
MapMonoid
groupBy TweetID
(TweetID, Map[URL, Long])

View Slide

object OuroborosJob {
def apply[P <: Platform[P]](source: Producer[P, ClientEvent], sink: P#Store[OuroborosKey, OuroborosValue]) =
source.filter(filterEvents(_))
.flatMap { event =>
val widgetDetails = event.getWidget_details
val referUrl: String = widgetDetails.getWidget_origin
val timestamp: Long = event.getLog_base.getTimestamp
val widgetFrameUrlOpt: Option[String] = Option(widgetDetails.getWidget_frame)
for {
tweetId: java.lang.Long <- javaToScalaSafe(event.getEvent_details.getItem_ids)
timeBucketOption: Option[TimeBucket] <- timeBucketsForTimestamp(timestamp)
} yield {
val urlHllOption = canonicalUrl(referUrl).map(hllMonoid.create(_))
val widgetFrameUrlsOption = widgetFrameUrlOpt map { widgetUrl: String =>
widgetFrameUrlsSmMonoid.create((referUrl, (widgetFrameUrlSetSmMonoid.create((widgetUrl, 1L)), 1L)))
}
val impressionsValue: OuroborosValue = RawImpressions(
impressions = 1L,
approxUniqueUrls = urlHllOption,
urlCounts = Some(embedCountSmMonoid.create((referUrl, 1L))),
urlDates = Some(embedDateSmMonoid.create((referUrl, timestamp))),
frameUrls = widgetFrameUrlsOption
).as[OuroborosValue]
Seq(
(OuroborosKey.ImpressionsKey(ImpressionsKey(tweetId.longValue, timeBucketOption)), impressionsValue),
(OuroborosKey.TopTweetsKey(TopTweetsKey(timeBucketOption)), topTweetsValue)
)
}
}.sumByKey(store)
.set(MonoidIsCommutative(true))
}

View Slide

.flatMap { event =>
for {
} yield {
}
impressions = 1L,
Seq(
)
}
}.sumByKey(store)
}
Filter Events

View Slide

.flatMap { event =>
for {
} yield {
}
impressions = 1L,
Seq(
)
}
}.sumByKey(store)
}
Filter Events
Generate KV Pairs

View Slide

.flatMap { event =>
for {
} yield {
}
impressions = 1L,
Seq(
)
}
}.sumByKey(store)
}
Filter Events
Generate KV Pairs
Sum into Store

View Slide

Brief Explanation
This job creates two types of keys:
1: ((TweetId, TimeBucket) => Map[URL, Impressions])
2: TimeBucket => Map[TweetId, Impressions]

View Slide

What Else?

View Slide


View Slide

What’s Next?

View Slide

- Akka, Spark, Tez Platforms
- More Spouts, Stores and Monoids
- Pluggable graph optimizations
- More tutorials!
Future Plans

View Slide

Open Source!

View Slide

•Summingbird is appropriate for the majority
of the real-time apps we have.
•Data scientists who are not familiar with
systems can deploy realtime systems.
•Systems engineers can reuse 90% of the
code (batch/realtime merging).
Summary

View Slide

Follow me at @sritchie
Thank You!

View Slide

Boston Storm Users: Summingbird, Scala and Storm

Boston Storm Users: Summingbird, Scala and Storm

Sam Ritchie

More Decks by Sam Ritchie

Other Decks in Programming

Featured

Transcript