Hadoop and Hubris

A short presentation introducing Apache Hadoop and describing what it is and some of the myths surrounding it. The talk is rounded off with a demonstration using Hadoop in the context of Microsoft HD Insight

Transcript

1. Hadoop and Hubris How Hadoop solves big data problems
2. What is it? • Hadoop is a project of the Apache Foundation • Apache Hadoop is an open-source software framework for storage and large-scale processing of data-sets on clusters of commodity hardware. Further resources at http://hadoop.apache.org/
3. What does it do... • Allow you to process large volumes of “unstructured data” All data has a structure, when talking about unstructured data people usually mean they want just part of the data, structured in a way they can make sense of. • Schema-on-Read not Schema-on-Write You can just load the data. You define a schema when you want to analyse it
4. What else does it do... • Allow you to distribute the processing load “Hadoop Distributed File System (HDFS) is designed to span large clusters of commodity servers. Data to be processed is spread across these servers. Jobs can then be distributed to crunch the data. Hadoop use the concept of a Name Node which controls and monitors the distribution and processing of data among any number of node servers • Allow you to extract the results of your processing Once you have crunched your data, you can view it in analysis and visualisation tools such as Power Query for Excel, Tableau and QlikView
5. What doesn’t it do... • Automagically process unstructured data Despite what many Snake Oil Salesmen would have you believe, there is no magic button that will turn raw data into business decisions with no effort on your part. • Remove the tedium of patiently mapping data There is no magic button that will discover and map the structure of your data. • Create Unicorns There is no magic button that will create Unicorns.
6. Some Hubris... Build a 360° View of the Customer • Retailers interact with customers across multiple channels, yet customer interaction and purchase data is often isolated in data silos. Few retailers can accurately correlate eventual customer purchases with marketing campaigns and online browsing behavior. (true) • Apache Hadoop gives retailers a 360° view of customer behavior. It lets them store data longer and identify phases of the customer lifecycle. (untrue) Better customer analytics increase sales, reduce inventory expenses and retain the best customers.(true) From http://hortonworks.com/industry/retail/
7. How it works...
8. Now you need some big data...
9. Demo... Create a Hadoop Cluster
10. Pre-processing Before you can do stuff in Hadoop you need to prepare your data. Hadoop likes CSV!
11. Pre-processing tools.. XML ⇒ Powershell works well JSON ⇒ Python is a good choice But you can use any language you like.
12. Upload your data... You need to upload your data to your servers In this case Azure blob storage
13. Create your tables and run your jobs... Now you can create tables and views based on your data
14. View your crunched data...