Introduction to Apache NiFi and Storm

Introduction to Apache NiFi and Storm

Transcript

1. Introduction to Apache NiFi & Storm Jungtaek Lim
2. WHO AM I? • Staff Software Engineer @ Hortonworks • remote worker • Open source prosumer • Committer of Jedis • PMC member of Apache Storm • Contributor of Apache (Spark, Zeppelin, Ambari, Calcite), Redis, and so on. • Contact • kabhwan@gmail.com • Twitter / LinkedIn / Github / Facebook • @heartsavior
3. Core Infrastructure Sources à Constrained à High-latency à Localized context à Hybrid – cloud / on-premises à Low-latency à Global context Regional Infrastructure DATA IN MOTION IN HORTONWORKS DATAFLOW (HDF) Source: http://ko.hortonworks.com/products/data-center/hdf/
4. What is Apache NiFi?
5. An easy to use, powerful, and reliable system to process and distribute data.
6. History of Apache NiFi
7. • Created by the United States National Security Agency (NSA) • originally named Niagarafiles • In 2014 the NSA submitted the source code to Apache Software Foundation, via the NSA Technology Transfer Program, entered incubation in December 2014 • Development of Apache NiFi continued at Onyara, Inc., a start up company • Became Apache Top-Level Project in July 2015 • Hortonworks acquired Onyara, Inc. in August 2015
8. Role of Apache NiFi
9. • Data acquisition and delivery • Simple transformation and data routing • Simple event processing • End to end provenance • Edge intelligence and bi-directional comms.
10. NOT intended to REPLACE 
 ‘distribute computation engines’ 
 (a.k.a streaming processing frameworks)
11. Features of Apache NiFi
12. Highly configurable • Loss tolerant vs guaranteed delivery • Low latency vs high throughput • Dynamic prioritization • Flow can be modified at runtime • Back pressure
13. More… • Designed for extension • Build your own processors and more • Secure • SSL, SSH, HTTPS, encrypted content, etc... • Multi-tenant authorization and internal authorization/policy management • MiNiFi subproject • Reduce footprint to ~ 40 MB
14. What is Apache Storm?
15. A free and open source distributed realtime computation system.
16. History of Apache Storm
17. Source: http://hortonworks.com/blog/brief-history-apache-storm/
18. Concepts of Apache Storm
19. • Spout: a source of streams in a topology • Bolt: a processing component which includes Sink • Stream: an unbounded sequence of tuples, defined with schema • Stream groupings: defines how that stream should be partitioned among the bolt's tasks • Topology: the logic for a realtime application represented to a DAG
20. Core vs Trident
21. Core Trident Computation Unit Record (tuple) Micro batch Latency Very low (sub-seconds) High (up to batch size) Similar to Spark Streaming Delivery Guarantee At least once Exactly once API Compositional Declarative Stateful Operator Supported from v1.0.0 Core feature
 (exactly-once) Windowing Time (processing time, event time), Count Tumbling window, Sliding window
22. Features of Apache Storm
23. • Supports number of connectors (17 connectors in master branch) • Automatic back-pressure • Distributed Cache • Flux (constructing topology via yaml) • Distributed Log Search • Dynamic Worker Profiling • Dynamic Log Levels • Topology Event Inspector • Resource Aware Scheduler • SQL (Experimental)
24. Future of Apache Storm Apache Storm 2.0 and beyond
25. • Clojure to Java translation • Unified Stream API with supporting exactly-once • Rework Metrics feature • Apache Beam runner • Streaming SQL with Apache Calcite • And more… • Performance • Usability
26. THANKS! Any questions?
27. Appendix A. Apache NiFi
29. NiFi EvaluateJsonPath / RouteOnAttribute configuration
30. NiFi PutHDFS / PublishKafka configuration
31. NiFi Queue options – Status History
32. NiFi Queue options – List queue
33. NiFi Data Provenance
34. Appendix B. Apache Storm
35. Distributed Log Search
36. Dynamic Worker Profiling
37. Dynamic Log Levels
38. Topology Event Inspector
39. Resource Aware Scheduler Source: Resource Aware Scheduling in Apache Storm, Hadoop Summit San Jose 2016