Lambda Architectures by Carlos Queiroz

Transcript

1. © 2014 SpringOne 2GX. All rights reserved. Do not distribute without permission. Implementing the Lambda Architecture with Spring XD By Carlos Queiroz
2. Me Advanced Field Engineer 2 Carlos Queiroz
3. Agenda ! • Introduction to the Lambda Architecture • Applying Lambda architecture to a business case • Implementation details 3
4. What is Lambda Architecture? Implementation of a set of desired properties on general purpose big data systems. ! Generic architecture addressing common requirements in big data applications. ! Set of design patterns for dealing with historical and operational data. 4
5. Approach is new, ideas not so… 5 Traditional / Relational Data Sources The Big Data Ecosystem Streaming Data Traditional Warehouse Analytics on Data at Rest Data Warehouse Analytics on Structured Data Analytics on
 Data in Motion MapReduce like models Non-Traditional /
 Non-Relational
 Data Sources Non-Traditional / Non-Relational Data Feeds Traditional / Relational 
 Data Sources Internet-
 Scale Data Sets Stream computing Hadoop
6. Why Lambda Architecture? To build a data system that can answer questions by running functions that take at the entire dataset as input. A general purpose data system 6
7. Desired Properties of such systems Fault-tolerance Generic Scale linearly, horizontally. Extensible Be able to achieve low latency updates when necessary. Be able to “ask” arbitrary questions to the system 7
8. How to support those properties? Decompose the problem into pieces 8 Speed Layer Batch Layer Serving Layer
9. Batch Layer 9 Master dataset Batch Layer Batch view Batch view Batch view runBatchLayer() { while(true) recomputeFunctions() }
10. The Master dataset 10 Master dataset Source of Truth Recom pute Write once Protect Read many
11. Master dataset requirements 11 Operation Requisite Comments Writes Efficient appends of new data Basically add new pieces of data. Easy to append new data Writes Scalable storage Need to handle possibly, petabytes of data Reads Support for parallel processing Functions usually work on the entire dataset. Need to support handling large amounts of data Reads Vertically partition data Not necessary to look all the data all the time. Some functions may need to look at only relevant data (e.g. 1 week of calls) Writes/Reads Costs for processing. Flexible storage Storage costs money (a lot). Need flexibility on how to store and compress data.
12. The Master dataset 12 HDFS - Hadoop Distributed File System
13. Serving Layer Makes the batch views “queryable” 13 Batch view Batch view Batch updates and random reads D istributed database
14. Desired properties on the Serving layer 14 Batch view Batch view High Throughput Low Latency
15. Serving layer database requirements 15 Batch writable Scalable Random reads Fault-tolerant
16. Batch and serving layers 16 Master dataset Batch Layer Batch view Batch view Batch view analytical functions, immutable storage Serving Layer pre-computed analytical functions (Views) Only property missing - low latency updates
17. Speed layer Allows arbitrary functions computed on arbitrary data on (near) real-time. 17 Feed Feed Feed Feed Feed Feed
18. Speed layer 18 RealTime Function RealTime Function RealTime Function RealTime Function RealTime Function RealTime Function RealTime View RealTime View RealTime View RealTime View RealTime View RealTime View RealTime View Narrow but more up to date view Depending of functions complexity incremental computation approach is recommended
19. Incremental computation realtime view = function(new data, realtime view) 19
20. Eventual Accuracy ! Some computations are harder to compute ! For such cases approximations are used. Results are approximate to the correct answer ! Sophisticated queries such as realtime machine learning are usually done with eventual accuracy. Too complex to be computed exactly. 20
21. Approximation & Randomisation ! ! Approximation find an answer correct within some factor (answer that is within 10% of correct result) ! Randomisation allow a small probability of failure (1 in 10,000 give the wrong answer) ! ! 21
22. Synopses structures • Sampling • Sketches • Histograms • Wavelets ! Library implementations • algebird (https://github.com/twitter/algebird) • samoa (https://github.com/yahoo/samoa/wiki) 22 http://charuaggarwal.net/synopsis.pdf
23. Stream processing (real-time function) Run the realtime functions to update the realtime views 23 Data Ingestion Stream Processing (realtime function)
24. One-at-a-time Divide your processing into worker processes, and put queues between the worker processes 24 Queues and workers paradigm
25. Generalised one-at-a-time approach • Works on a higher level • Stream computation defined as a graph (usually). • Storm, InfoSphere Streams models • Filters and pipes • Spring XD • At least once in case of failures. 25 cut -d" " -f1 < access.log | sort | uniq -c | sort -rn | less
26. Micro-batching • Small batches of events processed one at a time ! • Exactly one processing ! • Implementations: • Spark Streaming 26 batch #1 batch #2 Data Ingestion
27. Lambda Architecture 27 Ad-hoc queries analytical functions analytical functions, immutable storage raw data Data Ingestion Speed Layer Batch Layer Serving Layer raw data raw data raw data raw data pre-computed analytical functions
28. Principles of Lambda Architecture Store data in it’s rawest form Immutability and perpetuity ! Re-computation ! Query = function(all data) 28
29. Implementing the Lambda Architecture The ACM DEBS 2014 Grand Challenge1 29 To demonstrate the applicability of event-based systems to provide scalable, real-time analytics over high volume sensor data 1 http://www.cse.iitb.ac.in/debs2014/
30. ACM DEBS 2014 Challenge Application Scenario ! Analysis of energy consumption measurement 30 Short-term load forecasting makes load forecasts based on current load and what was learned over historical data Load statistics for real-time demand management finds outliers based on the energy consumption plug plug house_id dev_id plug_id
31. Data model 31 Data source Field Comments ID UNIQUE IDENTIFIER TIMESTAMP TIMESTAMP OF MEASUREMENT VALUE MEASUREMENT PROPERTY TYPE: 0 - WORK, 1 - LOAD PLUG_ID UNIQUE IDENTIFIER OF A PLUG HOUSEHOLD_ID UNIQUE IDENTIFIER OF A HOUSEHOLD HOUSE_ID UNIQUE IDENTIFIER OF A HOUSE Field Comments TS TIMESTAMP OF STARTING TIME PREDICTION HOUSE_ID HOUSE ID PREDICTED_LOAD PREDICTED LOAD PL - House Field Comments TS_START TIMESTAMP OF START TIME WINDOW TS_STOP TIMESTAMP OF STO PTIME WINDOW HOUSE_ID HOUSE ID PERCENTAGE % PLUGS LOAD HIGHER THAN NORMAL Outliers Field Comments TS TIMESTAMP OF STARTING TIME PREDICTION HOUSE_ID HOUSE ID HOUSEHOLD_ID PLUG_ID PREDICTED_LOAD PREDICTED LOAD PL - Plug
32. Analytical models 32 L ( si+2) = ( avgLoad ( si) + median ( avgLoad ( sj))) / 2 Load prediction per house, per plug Outliers For each house calculate the percentage of plugs which have a median load during the last hour greater than the median load of all plugs (in all households of all houses) during the last hour sj = s(i+2–n⇤k) It is based on the average load of the current window and the median of the average loads of windows covering the same time of all past days
33. How others did 33 Storm-based implementation
34. How others did 34 Oracle DB + Oracle Event Processing (OEP)
35. How others did 35 http://lsds.doc.ic.ac.uk/projects/seep/debs14-gc SEEP: Real-Time Stateful Big Data Processing
36. Implementation details Fork from https://github.com/pivotalsoftware/gfxd-demo 36 Thanks to Jens Deppe and William Markito
37. Our approach - ACM DEBS 2014 system architecture 37 RabbitMQ Spring XD Hadoop GemFire XD Web App (Spring boot) Batch layer Serving layer Speed layer Sensor Event Sensor Event Sensor Event Sensor Event Sensor Event
38. Data ingestion 38 Sensor Events RabbitMQ publishes Stream App Spring XD subscribes S S S S S S S S
39. Speed layer 39 Application flow Remove duplicates Filter Fix missing values transform Store to GFXD Sink Find outliers Tap Load Forecast Tap Load pred. model Processor Store to GFXD Sink Outliers Model Processor not a flow Store to GFXD Sink RT view RT view RT view RT view RT view RT view Load Forecast Tap Load pred. model Processor Store to GFXD Sink RT view RT view RT view P H
40. Stream actions 40 Duplicates filtering enrichment (compute time slice) compute model Produce RT View
41. Spring XD streams 41 pumpin - sends all data to a master queue sensoreventenricher - Consumes the data from the queue, filter and transform the data before store on HDFS findoutliers - Taps from master_ds stream to compute outlier model loadpred{h,p} - Taps from master_ds stream to compute load prediction for house and plug (2 streams) 5 streams
42. Batch Layer 42 Batch job that starts from Spring XD Uses cron to run every X hour/min/day? Load pred. model Job Hadoop Batch view Batch view Batch view Hadoop based system Store an immutable and constantly growing master dataset (of all datasets) ! Compute arbitrary functions (models) on the existing datasets. ! Essentially, runs the MR models.
43. Spring XD jobs • A unique job to run the MR model to compute the historical aspect of the models. • MR job runs every “day/hour/min” • Job is completely independent from the other streams 43 1 job Load raw_sensor table compute model update avg, outliers table
44. Serving layer (RT and Batch views) 44 Field Comments TS TIMESTAMP OF STARTING TIME PREDICTION HOUSE_ID HOUSE ID PREDICTED_LOAD PREDICTED LOAD Field Comments TS_START TIMESTAMP OF START TIME WINDOW TS_STOP TIMESTAMP OF STO PTIME WINDOW HOUSE_ID HOUSE ID PERCENTAGE % PLUGS LOAD HIGHER THAN NORMAL Gemfire XD Get updates from MR and SP models Holds both batch and real-time views Field Comments TS TIMESTAMP OF STARTING TIME PREDICTION HOUSE_ID HOUSE ID HOUSEHOLD_ID PLUG_ID PREDICTED_LOAD PREDICTED LOAD
45. Web UI 45 Web App (Spring boot) view view view
46. Why Gemfire XD? ‣ Distributed database ! ‣ Tightly integrated with Pivotal Hadoop ! ‣ SQL support ! ‣ Fault-tolerant ! ‣ In-memory (fast access) 46 GemFire XD Hadoop column_name data_type column_name data_type Model: Data has_many_and_belongs_to_many :teams Table1 column_name data_type column_name data_type Model: Data has_many_and_belongs_to_many :teams Table1 column_name data_type column_name data_type Model: Data has_many_and_belongs_to_many :teams Table1 column_name data_type column_name data_type Model: Data has_many_and_belongs_to_many :teams Table1 column_name data_type column_name data_type Model: Data has_many_and_belongs_to_many :teams Table1 column_name data_type column_name data_type Model: Data has_many_and_belongs_to_many :teams Table1
47. Why Pivotal HD (PHD) 47 ‣ Hadoop based ! ‣ Tightly integrated with Gemfire XD GemFire XD Hadoop column_name data_type column_name data_type Model: Data has_many_and_belongs_to_many :teams Table1 column_name data_type column_name data_type Model: Data has_many_and_belongs_to_many :teams Table1 column_name data_type column_name data_type Model: Data has_many_and_belongs_to_many :teams Table1 column_name data_type column_name data_type Model: Data has_many_and_belongs_to_many :teams Table1 column_name data_type column_name data_type Model: Data has_many_and_belongs_to_many :teams Table1 column_name data_type column_name data_type Model: Data has_many_and_belongs_to_many :teams Table1
48. Why Spring XD? ‣ Data Ingestion ! ‣ Real-time Analytics ! ‣ Workflow Orchestration ! ‣ Integration 48
49. Full Open source implementation 49 HBase?? Hadoop SpringXD
50. Is Lambda Architecture perfect? • Suitable for specific use cases • Doesn't contemplate reference data access. • Not always possible to use same model for Real-time and Batch • Other ideas to improve the lambda architecture • Multiple batch layers • incremental batch layers 50
51. Source code 51 https://bitbucket.org/caxqueiroz/debs2014-challenge
52. Thank you!! 52