IoE: Using Elasticsearch to Unlock an Analytical Goldmine

This talk was presented at the inaugural Elastic{ON} conference, http://elasticon.com

Session Abstract:

The true value of IoE starts to be realized not when you connect things, but when you are best able to use the data generated by those things. Not only is the final impact of a device event important, but the ability to analyze the cascading effects felt throughout the device ecosystem and different layers of the business is paramount. The Cisco Analytics Incubation Center developed a real time, data-driven, and rules-based framework to identify these cause-effect relationships. This talk will showcase how Elasticsearch has helped make it easy to utilize IoE data in real-time, and how critical it is to do so given the flood of IoE device data expected in the future.

Presented by Dr. Connie Brett & Vikram Kalyanaraman, Cisco

Transcript

1. IoE: Using Elasticsearch to Unlock an Analytical Goldmine Dr. Connie Brett and Vikram Kalyanaraman
2. { } CC-BY-ND 4.0 2   Source: Cisco Consulting Services primary research, 2013 $6.4T Of  IoE  Opportunity  over  the  Next  10  Years  will  be  Realized  Through  Data  &  AnalyDcs   Data Monetization Customer Engagement Optimized Operations Threat Minimization Organizational Effectiveness
3. { } CC-BY-ND 4.0 3   Mike  O'Callaghan–Pat  Tillman  Memorial  Bridge   Business   Enablers   things people process data
4. { } CC-BY-ND 4.0 4   { } CC-BY-ND 4.0 What’s the state of my empire? ATM not working Sorry for the inconvenience All good Fine Umm … ATMs are down in SF Why are they down? Not sure Day 1
5. { } CC-BY-ND 4.0 5   { } CC-BY-ND 4.0 Let’s use IoE We need to know sooner We’ll build a dashboard Here’s some data from sensors We’ll crunch it offline and aggregate results Day 2
6. { } CC-BY-ND 4.0 6   { } CC-BY-ND 4.0 What’s the state of my empire? How does that data help? We built this dashboard Not sure Are any ATMs down? The average uptime for our network is 247,365 mins Day 3
7. { } CC-BY-ND 4.0 7   Mike  O'Callaghan–Pat  Tillman  Memorial  Bridge   Business   Enablers   things people process data
8. { } CC-BY-ND 4.0 GOALS 8   Build  a  mechanism  to  connect  relevant  data  from   devices  and  things  to  business  outcomes   1 Model  and  visualize  cause-­‐effect  rela3onships   across  IT  &  Business  layers  in  near  real-­‐Dme   2 Enable  rapid  root  cause  analysis  of  IT  issues   that  affect  the  business   3
9. { } CC-BY-ND 4.0 9   Development Scenario: A large retailer with 100s of stores globally
10. { } CC-BY-ND 4.0 Challenges 10   { } CC-BY-ND 4.0         Volume   Types   Status   Data  Variety   Data  Velocity   Business  Value??     S E N S O R S   DATA   Sta3c  &  Highly   Aggregated  Dashboards   Delayed  Insights  from   Tradi3onal  Processing   VISUALIZATION  
11. { } CC-BY-ND 4.0 Data & Workflow 11   { } CC-BY-ND 4.0 AggregaDons   AnalyDcs   Business  Rule  ApplicaDons   Filtering   Continual Analysis Engine Alerts & Events x x
12. { } CC-BY-ND 4.0 Data & Workflow 12   { } CC-BY-ND 4.0 AggregaDons   AnalyDcs   Business  Rule  ApplicaDons   Filtering   Continual Analysis Engine Alerts & Events data  driven   visualiza3on   mongo  connector  
13. { } CC-BY-ND 4.0 Visualization Business   Layers   Device   Data   Business Rules
14. { } CC-BY-ND 4.0 Cause-Effect Visualization RULES  ENGINE   connects     CONCRETE  EVENTS   with     BUSINESS  HIERARCHIES   Ingest  Events   &  Alerts   Run   Rules   Propagate  effect  through  tree  using   Color  to  indicate  severity  of  effect   >   >  
15. { } CC-BY-ND 4.0 15   LIVE DEMO
16. { } CC-BY-ND 4.0 16   END OF DEMO
17. { } CC-BY-ND 4.0 17   { } CC-BY-ND 4.0 ATM not working Sorry for the inconvenience. We are aware of the issue and are working to resolve it. In the meanwhile, the nearest working ATM is just one block from your current location. Day 4 >   ATM 1 DOWN for 3 mins ATM 5 UP (nearest) What’s the state of my empire? We have current data on an outage of a few ATMs in SF. Customers are being redirected to nearby ATMs. We know the issue is localized and therefore impact to business is minimal.
18. { } CC-BY-ND 4.0 The Causality Framework 18   LIGHTWEIGHT   EASILY  CUSTOMIZED   LOW  MAINTENANCE   We  are  embedding  the  Causality   Framework  in  our  Connected   AnalyDcs  Products  and  SoluDons  
19. { } CC-BY-ND 4.0 19   Q & A
20. { } Special Thanks To: Dr. Connie Brett cbrett @cisco.com @cbdsabc Vikram Kalyanaraman vkalyana @cisco.com @vikalytics
21. { } CC-BY-ND 4.0 This work is licensed under the Creative Commons Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA { 21