SOA & Big data

Some notes on combining SOA approach with big data

Transcript

1. SOA & Big data   Arnon  Rotem-­‐Gal-­‐Oz  
2. Sept  2012  –  iOS6  launched  with  new  maps  applica>on  
3. But  something  went  terribly  wrong….   hEp://theamazingios6maps.tumblr.com/  
4. •  It  isn’t  just  about   geKng  all  the  data   there   •  Algorithms  are  cool   but  we  need  humans   in  the  loop   •  Hire  the  right  people   •  Test  !  Test  !  Test!     hEp://theamazingios6maps.tumblr.com/  
5. hEp://theamazingios6maps.tumblr.com/   It  isn’t  just  one  pile  of  data  
6. Integra>ng  Big  data  &  SOA     Yoel  Ben  Avraham  -­‐  hEp://www.flickr.com/photos/epublicist/3546059144/  
8. Data     Refinery     Ofer  Berger     hEp://www.haifacity.com/allsites/allpic/a/A1738/A1738Pic3326.jpg  
10. ETL  integra>on   DB  integra>on   File-­‐based  integra>on   Online  integra>on   Department   Server   DB  
11. ASB BLT HDL AFT TGI FRY DRW SWG QYD DLY BST WIU ASB ZIS XOI CUI RMO DLY XPS KYF KFC WHR JIA GEX FQA VUH HCO WKD ECP SKD MFP WCP DKE AJT   Object  soup  
12. ASB BLT HDL AFT TGI FRY DRW SWG QYD DLY BST WIU ASB ZIS XOI CUI RMO DLY XPS KYF KFC WHR JIA GEX FQA VUH HCO WKD ECP SKD MFP WCP DKE AJT   Services   Invoices Customer Promotions Orders
13. Service   Describes   Endpoint   Exposes   Messages   Sends/receives   Contracts   Binds  to   Service     consumer   Implements   Policy   Governed  by   Sends/receives   Adheres  to   Component   Rela>on   Key   Understands   Serves  
14. Interac>ons   Customer   Categories   Agents  
15. Integra>ng  Big  data  &  SOA     Yoel  Ben  Avraham  -­‐  hEp://www.flickr.com/photos/epublicist/3546059144/  
16.                                                                 Coordinator*   Prepare/commit/undo   Service  consumer   Protocol   Rela>on   Key   SOA  component   PaEern  component   Concern/aEribute     RegistraDon   Perform     acDvity   Compensate   Create   context   Ini>ator   Service     Par>cipator   Perform     acDvity   Compensate   Prepare  /   commit  /   undo   Register   AcDviDes  and  replies   AcDviDes  and  replies   Saga  
17. Hadoop Cluster NIM Interaction Recordings ETL Customer HBase Raw (HDFS) Interactions HBase Data Management HBase HCatalog Resolved Interactions(H DFS) Categories HBase HBase Resolved Interactions(H DFS)
18. So,   what’s  the     problem  ?  
19.  &  Big  data     can’t  move  
20. Performance  of  joins  in  distributed   system  sucks!   Node 1 customers A-H Interactions 0-99 Node 2 customers I-M Interactions 100-199 Node 3 customers N-Z Interactions 200-299 {”Interac>on":  {      "id":  ”5",        ”par>cipants":  {          ”customer":  [              {”surname":  ”McDonalds",  ”name":  ”Old"},]      }   }}  
21. Cookie  cuEer   scalability    
22. Cell  architecture   Node   2   Node   3   Node   1   Node   N  
23. Cell  Architecture   BUS Categories Customers Interactions Reference Data ORCA … HBase HDFS HBase HBase HBase HBase
24.                   Initiate business process Workflow engine Endpoint   Workflow instance Invoke services Manage   process   Route request Host   workflows   Schedule   Service                   Endpoint   Service Manage   workflows   Monitor   workflows   Orchestra>on  
25. Map  Reduce  processing  pipeline   Resolve Customer IDs (Custoemr) Categorize Segment (Categorization) Update Segment document (Interaction) Map pipeline Segment Row Retrive segment data - create segment document (Interaction) Write Categories Results (Categorization) Write Interaction (interaction) Customers Local cache InteractionID, Segment Row Map Prepare data mart Export (Datamart) Update Interaction document (Interaction) Reduce pipeline Interaction & Segments Categorize Interaction (Categorization) Write Categories Results (Categorization) Write Interaction (interaction) Reduce Write Interaction (interaction) Hadoop Map/Reduce
26. Map  Reduce  processing  pipeline   Resolve Customer IDs (Custoemr) Categorize Segment (Categorization) Update Segment document (Interaction) Map pipeline Segment Row Retrive segment data - create segment document (Interaction) Write Categories Results (Categorization) Write Interaction (interaction) Customers Local cache InteractionID, Segment Row Map
27. Data  Facets  
28. In-memory Data grid Columnar Graph Indexing NewSQL Columnar Caching HBase   Hypertable   Neo4j   Apache  Solr   AKvio   IndexTank   RavenDB   Cassandra   MongoDB   CouchDB   ScaleBase   VoltDB   Amazon  RDS   HP  Ver>ca   EMC  Greenplum   IBM  Netezza   Microsoo  PDW   Aster  Data   ParAccel   Memcached   GigaSpaces   Redis   GridGain   Oracle  Coherence   WebSphere  eXtreme  Scale   Pregel   Hama   SAP  HANA   Oracle  Exadata   Accumulo   Document Relational Analytics/MPP Key-value store Distributed file systems Hadoop   GlusterFS  
29. Datawarehouse   (Hadoop/Hbase)         20  years     detailed   aggregated       Datamart(s)   (RDBMS)       6-­‐12  months   Detailed     1-­‐3  years  aggregated   Cube   (MOLAP)       6-­‐?  Months   aggregated   Real-­‐>me   (in  memory)   1-­‐7  days   detailed   Data  is  mul>-­‐>ered  
30. Data  warehouse   (Hadoop/Hbase)         20  years     detailed   aggregated       Real-­‐>me     1-­‐7  days   detailed   Datamart(s)   (Columnar)     6-­‐12  months   Detailed     Data  is  mul>-­‐>ered  
31. SOA  leaves  us  with  a  lot   of  isolated  data  
32.                                             Subscribed/ pulled data                                           Pull data Data backend Endpoint Out Load Report Ingest   Clean   Join   Transform Transpose   Produce     reports   Report Endpoint Request Raw  data   ODS/DM                           SQL endpoint                           SQL endpoint Landing  area   Service Aggregated  Repor>ng  
33.  Landing   Raw  data           DW/ODS   Views   Transforma>on   service   1 1 2 3 4 5 Load   service   2 Report   service  
34. Report tool Data mart 4 3 Raw data (HDFS) Aggregation map/reduce HBase ETL (map/reduce +ETL) Drill through REST API Details Aggregates 1 2 2 5 6 7 8 9 10
35. Take  aways   SOA  &  Big  data  are  beEer  together  
36. Arnon  Rotem-­‐Gal-­‐Oz     arnonr@nice.com     hEp://www.nice.com     hEp://arnon.me/soa-­‐paEerns     arnon@rgoarchitects.com     hEp://arnon.me     @arnonrgo