StormTrooperStreams.pdf

Processing the unbounded streams of data in a distributed system sounds like a challenge. Fortunately, there is a tool that can make your way easier. Łukasz will share his experience as a "Storm Trooper", a user of Apache Storm framework, announced to be a first streaming engine to break the 1-microsecond latency barrier. His story will start by describing the processing model. He'll tell you how to build your distributed application using spouts, bolts, and topologies. Then he'll move to components that make your apps work in a distributed way. That's the part when three guys: Nimbuses, Supervisors, and Zookeepers join in and help to build a cluster. As a result, he'll be able to show you a demo app, running on Apache Storm. As you know, Storm Troopers are famous for missing targets. Łukasz will sum up the talk by sharing the drawbacks and ideas that he missed when he first met this technology. After the presentation, you can start playing with processing streams or compare your current approach with the Apache Storm model. And who knows, maybe you'll become a Storm Trooper.

Transcript

1. The Storm Trooper Way of Processing Streams 10.06.2020 J on The beach Post apocalypse session By Łukasz Gebel
2. A long time ago, in a galaxy far, far away...
3. A long time ago, in a galaxy far, far away...
6. Agenda • What is Apache Storm? • Processing Model • Experience
7. Apache Storm “(...) distributed realtime computation system.“ storm.apache.org
8. Distributed • Nodes communicate to achieve a goal • Tasks distribution focus
9. Realtime • Finish processing before deadline • “a million tuples processed per second per node”
10. Processing Model RECRUIT DISPATCH
11. Stormtroopers
12. Stormtroopers
13. Tuple • Set of named fields and values • Single chunk of data stream • Any serializable object
14. Stream Unbounded sequence of tuples . . . . . .
15. Spout • Starting point of a pipeline • Can produce data • Can poll data from external sources
16. Spout • Reliable • Not reliable
17. Bolt • Tuples processor • Execute work on input tuple • May produce output tuples
18. Messaging guarantees • Best effort
19. Messaging guarantees • Best effort • At least once
20. Messaging guarantees • Best effort • At least once • Exactly once
21. Messaging guarantees • Best effort • At least once • Exactly once - Trident
22. Messaging guarantees • Tuple tree • ACK or FAIL signals
23. Tuple tree SPOUT
24. Tuple tree SPOUT BOLT 1
25. Tuple tree SPOUT ACK BOLT 3
26. Tuple tree SPOUT BOLT 3
27. Tuple tree SPOUT ACK
28. Tuple tree SPOUT
29. Tuple tree SPOUT - ACK
30. Tuple tree SPOUT BOLT 1 FAIL
31. Tuple tree SPOUT - FAIL
32. Tuple tree SPOUT BOLT 1 TUPLE LOST ?
33. Tuple tree • Message timeout SPOUT - FAIL
34. Topology • Distributed application • Set of Spouts & Bolts • Directed Acyclic Graph
35. Topology
36. Code!
37. Cluster
38. Nimbus • Distributes topologies on cluster • Sends orders to: – Start – Stop – Kill
39. Supervisor • Receives orders and run workers • Workers are JVMs • Spouts & Bolts are tasks
40. Zookeepers • Cluster coordination • 2N + 1 instances
41. Let’s run the code!
42. Experience
43. Light Side • Easy to start • Flexible • Easily integrates with other tools • Minimal performance overhead • Out of the box distributed architecture
44. Dark Side • Topologies are static • Scaling – run multiple topologies or rebalance (it may stop for a moment!) • When task fail whole worker is restarted • Mind serialization and network latency while designing topology
45. Summary • We know how to build topologies • We know how to run them on cluster • We know the main pros and cons
46. Call to action
47. Call to action • If your new to distributed processing start your journey with Storm • If your experienced – benchmark its performance • You can get managed cloud Storm cluster on Azure, using HDInsight
49. Bibliography • Code: https://github.com/rauluka/stormtrooper-streams • Presentation: https://speakerdeck.com/rauluka7/stormtrooperstreams • https://storm.apache.org/releases/2.1.0/Tutorial.html • https://www.cloudera.com/tutorials/storm-in-trucking-iot/3.html • https://azure.microsoft.com/en-us/services/hdinsight/ • https://github.com/yahoo/streaming-benchmarks • https://storm.apache.org/releases/2.1.0/Trident-tutorial.html • https://storm.apache.org/releases/2.1.0/Guaranteeing-message-processing.html • https://www.cloudera.com/products/open-source/apache-hadoop/apache-storm.html • https://dzone.com/articles/apache-spark-vs-apache-storm • https://docs.microsoft.com/en-us/azure/hdinsight/storm/apache-storm-quickstart
50. Graphics • https://pixabay.com/photos/stormtrooper-star-wars-lego-storm-2899993/ • https://pixabay.com/photos/stormtrooper-lego-push-effort-4601509/ • https://pixabay.com/photos/stormtrooper-star-wars-lego-storm-2899982/ • https://pixabay.com/photos/stormtrooper-skateboard-lego-1995015/ • https://pixabay.com/photos/stormtrooper-star-wars-lego-storm-1343877/ • https://pixabay.com/photos/office-people-accused-accusing-2539844/ • https://pixabay.com/illustrations/ship-cosmos-space-technology-3857479/ • https://pixabay.com/photos/star-wars-storm-trooper-costume-2592430/ • https://i.ytimg.com/vi/Yv93L1KqMMc/maxresdefault.jpg • https://2.allegroimg.com/s1024/0cc3cf/5792fb7d49229863ad2b51536bb2 • https://pm1.narvii.com/6287/eaeff089928ea166e5987e2169201437459488fc_hq.jpg • https://pixabay.com/photos/star-wars-darth-wader-villain-2463926/ • https://pixabay.com/photos/stormtrooper-star-wars-lego-storm-1343772/
51. Q & A
52. Bibliography • Code: https://github.com/rauluka/stormtrooper-streams • Presentation: https://speakerdeck.com/rauluka7/stormtrooperstreams • https://storm.apache.org/releases/2.1.0/Tutorial.html • https://www.cloudera.com/tutorials/storm-in-trucking-iot/3.html • https://azure.microsoft.com/en-us/services/hdinsight/ • https://github.com/yahoo/streaming-benchmarks • https://storm.apache.org/releases/2.1.0/Trident-tutorial.html • https://storm.apache.org/releases/2.1.0/Guaranteeing-message-processing.html • https://www.cloudera.com/products/open-source/apache-hadoop/apache-storm.html • https://dzone.com/articles/apache-spark-vs-apache-storm • https://docs.microsoft.com/en-us/azure/hdinsight/storm/apache-storm-quickstart