Distributed Cache API Ã Topology resources – DicJonaries, ML Models, GeolocaJon Data, etc. Ã Typically packaged in topology jar – Fine for small files – Large files negaJvely impact topology startup Jme – Immutable: Changes require repackaging and deployment
Distributed Cache API à Allows sharing of files (BLOBs) among topologies à Files can be updated from the command line à Allows for files from several KB to several GB in size à Files can change over the lifeJme of the topology à Allows for compression (e.g. zip, tar, gzip)
Distributed Cache API à Two implementaJons: LocalFsBlobStore and HdfsBlobStore à Local implementaJon supports ReplicaJon Factor (not needed for HDFS-backed implementaJon) à Both support ACLs
Distributed Cache API CreaJng a blob: – storm blobstore create --file dict.txt --acl o::rwa --repl-fctr 2 key1 Making it available to a topology: – storm jar topo.jar my.topo.Class test_topo -c topology.blobstore.map=‘{"key1":{"localname":"dict.txt", "uncompress":"false"}}'
Nimbus HA à Increase overall availability of Nimbus à Nimbus hosts can join/leave at any Jme à Leverages Distributed Cache API à Topology JAR, Config, and Serialized Topology uploaded to Distributed Cache à ReplicaJon guarantees availability of all files
AutomaBc Back Pressure à In previous Storm versions, the only way to throFle topologies was to enable ACKing and set topology.spout.max.pending. à If you don’t require at-least-once guarantees, this imposed a significant performance penalty.
AutomaBc Back Pressure à High/Low Watermarks (expressed as % of task’s buffer size) à Back Pressure thread monitors task’s buffers à If High Watermark reached, slow down Spouts à If Low Watermark reached, stop throFling à All Spouts Supported
Resource Aware Scheduler à Specify the resource requirements (Memory/CPU) for individual topology components (Spouts/Bolts) à Memory: On-Heap / Off-Heap (if off-heap is used) à CPU: Point system based on number of cores à Resource requirements are per component instance (parallelism maFers)
Resource Aware Scheduler à CPU and Memory availability described in storm.yaml on each supervisor node. E.g.: supervisor.memory.capacity.mb: 3072.0 supervisor.cpu.capacity: 400.0 à ConvenJon for CPU capacity is to use 100 for each CPU core
Resource Aware Scheduler SpoutDeclarer spout = builder.setSpout("sp1", new TestSpout(), 10); //set cpu requirement spout.setCPULoad(20); //set onheap and o2eap memory requirement spout.setMemoryLoad(64, 16); BoltDeclarer bolt1 = builder.setBolt("b1", new MyBolt(), 3).shuffleGrouping("sp1"); //sets cpu requirement. Not neccessary to set both CPU and memory. //For requirements not set, a default value will be used bolt1.setCPULoad(15); BoltDeclarer bolt2 = builder.setBolt("b2", new MyBolt(), 2).shuffleGrouping("b1"); bolt2.setMemoryLoad(100);
NaBve windowing support à Fundamental abstracJon for processing conJnuous streams of events à Breaks the stream of events into sub-sets à The sub-sets can then be aggregated, joined or analyzed. Why is it important?
NaBve windowing support à Trending items – The top trending search queries in the last day – Top tweets in the last 1 hour à Complex event processing – Windowing is fundamental to gain meaningful insights from a stream of events – E.g. If same credit card is used from geographically distributed locaJons within the last one hour, it could be a fraud. Some use cases
NaBve windowing support public class WindowMaxBolt extends BaseWindowedBolt { private OutputCollector collector; public void prepare(Map conf, TopologyContext context, OutputCollector collector) { this.collector = collector; } public void execute(TupleWindow inputWindow) { int max = Integer.MIN_VALUE; for (Tuple tuple: inputWindow.get()) { max = Math.Max(max, tuple.getIntegerByField(”val")); } collector.emit(new Values(max)); } public staBc void main(String[] args) { // ... builder.setBolt("slidingMax", new WindowMaxBolt().withWindow(DuraJon.minutes(10), DuraJon.minutes(2))); // submit topology StormSubmiFer.submitTopologyWithProgressBar(args[0], conf, builder.createTopology()); } } triggered when the window ac.vates based on the window configura.on
NaBve windowing support à Processing Jme à Event Jme – e.g. Processing events out of a log file based on when the event actually happened – Implemented based on watermarks à Out of order and late events – e.g. Mobile gaming data where the user goes offline and data arrives awer someJme Windowing consideraBons 6:02 6:01 6:05 6:08 7:01 7:00 6:10 6:12 7:02 6:15 Processing Jme 6:09 6:12 6:08 watermark watermark 6:10 late event
State Management à Allows storm bolts to store and retrieve the state of its computaJon à The framework automaJcally and periodically snapshots the state of the bolts à In-memory and Redis based state stores à ImplementaJon based on – Asynchronous Barrier Snapsholng (ABS) algorithm [1] – Chandy-Lamport Algorithm [2] 1. hFp://arxiv.org/abs/1506.08603 2. hFp://research.microsow.com/en-us/um/people/lamport/pubs/chandy.pdf
State Management à CheckpoinJng/Snapsholng: What you get Spout Checkpoint Spout Stateful Bolt1 Stateful Bolt2 Bolt1 State Store ACKER ACK ACK ACK chkpt chkpt chkpt
Streaming SQL à SQL is a well adopted standard – available in several projects like Drill, Hive, Phoenix, Spark à Faster development cycle à Opportunity to unify batch and stream data analyJcs Benefits SELECT word, COUNT(*) AS total FROM words GROUP BY word
Streaming SQL Ã Example CREATE EXTERNAL TABLE ORDERS (ID INT PRIMARY KEY, UNIT_PRICE INT, QUANTITY INT) LOCATION 'kafka://localhost:2181/brokers?topic=orders’ CREATE EXTERNAL TABLE LARGE_ORDERS (ID INT PRIMARY KEY, TOTAL INT) LOCATION 'kafka://localhost:2181/brokers?topic=large_orders’ INSERT INTO LARGE_ORDERS SELECT ID, UNIT_PRICE * QUANTITY AS TOTAL FROM ORDERS WHERE UNIT_PRICE * QUANTITY > 50
Apache Beam (incubaBng) IntegraBon à Unified API for dealing with bounded/ unbounded data sources (i.e. batch/ streaming) à One API. MulJple implementaJons (execuJon engines). Called “Runners” in Beamspeak.
gionn
takuroabe
akitoshiga
pfn
camiha
satoshi7
takahashim
moznion
taketakekaho
yazm
yuuuuuuu168
akhino
matthewcrist
ryanjones
codingconduct
csswizardry
addyosmani
eileencodes
aleyda
cassininazir
lara
nikkihalliwell
tmiket
