• Run stateless services such as Web server, app server, etc. and Big Data services like HDFS, C*, Spark, etc. together on one cluster • Dynamic partitioning of your cluster, depending on your needs (business requirements) • Increased utilization (10% → 80% an more)
source – Convenient way to regulate data flow • Frequency counting as a simple task – Demonstrates the correctness of the framework, not accidently benchmarking C++ vs. Java, etc. – Dictionary limited to 9000 words to avoid excess memory allocation / pressure • Sample msg throughput for data source & sink • End-to-end latency (Concord only)
• 3 operator topology, a => b => c • a reads from a queue • b counts words – with every tuple, updated count emitted down stream • c writes the result into a log file as CSV plaintext – word, frequency • Log files to be post processed to determine accuracy
systems • Measuring latency is tricky – Requires instrumentation – The ability to follow a message all the way through the processing pipeline • Necessary to isolate Kafka consumer performance
• Scale up the Kafka cluster • Isolate the performance of Kafka consumers • Optimization efforts • Other frameworks like Samza, Flink, etc. • Instrument non-Concord frameworks with tracing to measure end-to-end latency
mhausenblas
plaidtech
shift_evolve
shinya337
kaonavi
taishin
kou_kita
mntsq
ahus1
yhana
ren510dev
masahirokawahara
nori_shinoda
morganepeng
eileencodes
lara
orderedlist
tammyeverts
addyosmani
bkeepers
kastner
aleyda
chriscoyier
bluesmoon
