What Does Big Data Really Mean for Your Business?

Presented by Leslie Hawthorn at the All Things Open 2014 conference:.

Abstract:
Today, people are putting massive amounts of data into a sink somewhere and then applying structure and analysis later, and getting insights sometime after that. Then, with each new data format / type, that leads to writing more code (and more waiting for actionable insights). Systems are becoming easier to use, but we’re getting very little business value from the data we’re stockpiling.

Elasticsearch has a mission to make any and all data (structure or unstructured) readable and usable for businesses everywhere, in real time.

Data collection and analysis can and does solve real business problems when sliced and diced in ways that make sense. Open Source technologies are vital to how we approach these data challenges, and bring us closer to the moonshot promises we’re making about the power of “big data.” Elasticsearch built an open source solution to solve complex problems that were previously dumped onto data scientists, and made insights available and readable across the company.

In her talk, Leslie Hawthorn will introduce the technical features of the Elasticsearch ELK stack against the backdrop of the Open Source world, with an eye towards building a growing business that’s changing the way people look at their enterprise foundations and what companies should expect from their vendors and technology partners. In addition, Leslie will touch upon the importance of fostering an open source community to provide valuable feedback to your development team and make your users’ lives easier.

Transcript

1. What Does Big Data Really Mean for your Business? Leslie Hawthorn Director of Developer Relations leslie.hawthorn@elasticsearch.com @lhawthorn
3. Even our small data has “Big Data”
4. Every day we create more… User Generated Machine Generated •  Tweets •  Blogs •  Books •  Notes •  Emails •  Comments •  Logs •  Events •  Statistics •  Alerts •  Metrics
5. And it’s growing… 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 2010 2016 Exabytes Year machine social voip enterprise
6. But there is a gap Storing Data Actionable Insight
7. Data From Any Source Instantly Analyze Actionable Insights The promise of “Big Data”
8. Massive effort in cleaning the data. Takes weeks to analyze. Insights come too late. The reality “I’m a data janitor.” Senior Director of Data Science, major Apache Hadoop distro
9. Why?
10. Balancing Structure and Scale SQL   Rela)onships   Structure   Can  it  Scale?   SQL   Yes   Yes   No   NoSQL   No   Yes   Yes   Hadoop   No   No   Yes   We got excited about Scale, but now suffer for lack of Structure. Which means: We can store a lot of data, but it takes a ton of work to search, analyze and get insights out of it. Hadoop Also Brings: Intensive Engineering and a Large Upfront Investment
11. Where we’re going
12. Your Big Data pipeline •  Incredibly crowded landscape in both the open source and proprietary worlds •  Most big data tools were built for people with Massive Data •  Tools now focus on making working with both structured & unstructured data easier
13. Why open source? http://www.flickr.com/photos/16462767@N00/3984507322/ flexibility
14. Data From Any Source Instantly Store, Search & Analyze Actionable Insights Elasticsearch: Search, Analytics, Logging & Visualization Logstash Elasticsearch Kibana The ELK Stack
15. Data Democratization
16. Big Data for our business •  Measuring our community health metrics Demo time…. [see appendix for screenshots from demo]
17. Be a data superhero, not a janitor!
18. Questions? Leslie Hawthorn Director of Developer Relations leslie.hawthorn@elasticsearch.com @lhawthorn
19. Appendix – Demo Slides