Phoenix Data Conference 2014 - Raz Yalov

Sometime grandiose words such as “BIG-DATA” are blocking us from checking if this is the a technology our company should use. Correct, not everyone is Google/Facebook/Twitter but this doesn’t make your few Tera-bytes of data less important than the MANY Peta-bytes of data that these monsters are dealing with. In a world where each desktop can hold more than a TB of data, your traditional tool set might be limiting your options. Learn how to ease in your business’ transition into the Big-Data world so that your addition of it is more of a “walk in the park” thank a stressful “marathon”

Transcript

1. © 41st Parameter. All Rights Reserved. CONFIDENTIAL Small step for your company, Big (data) step for your customers Raz  Yalov   20141025T113000-­‐0700  
2. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 2
3. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 3
4. © 41st Parameter. All Rights Reserved. CONFIDENTIAL So what is BIG data?
6. How Big is Big Data?
7. How Big is Big Data? ONLY AS BIG AS YOU NEED IT TO BE!
8. What is BIG data? Usually >= Petabytes
10. What is BIG data? Requires sampling/ filtering and/or M-R
11. © 41st Parameter. All Rights Reserved. CONFIDENTIAL How BIG data is being used by 41st Parameter?
12. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 12 Fraud Usage Analyze Performance of Risk Systems
13. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 13 Online Marketing Analyze Customer Provided Data
14. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 14
15. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 15 In-­‐House   Amazon  S3   both   §  4-­‐5  HDFS  nodes   §  Physical  servers   §  SSD  disks   §  Customer  uploaded   data   §  Large  scale   §  Low  Risk  
16. .lzo + .lzo.index (snappy – coming soon)
17. S3 is: §  Dynamic §  on-demand §  Scalable §  cheap §  Public
18. BUT
19. §  Sensitive data §  EMR slow to launch §  On-Demand… Not
21. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 21 A few Data Scientists (always looking for more)
29. R U Ready?
31. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 31 Securing Data
32. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 32
33. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 33 M-R Is NOT Easy!
34. HIVE Is NOT SQL
35. PIG Is NOT SQL
36. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 36 Pig IS Bacon!
37. M-R MUST be Tested
38. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 38 Don’t trust your mind
39. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 39
41. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 41
42. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 42 Validate your data & assumptions
43. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 43
44. ?  
45. © 41st Parameter. All Rights Reserved. CONFIDENTIAL Thank you for your time Raz Yalov CTO www.the41st.com ryalov@the41st.com 480.776.5506 direct 480.313.1359 mobile 16260 N. 71st Street Suite 400 Scottsdale AZ 85254
47. © 41st Parameter. All Rights Reserved. CONFIDENTIAL Small step for your company, Big (data) step for your customers Raz  Yalov   20141025T123000-­‐0700  
48. © 41st Parameter. All Rights Reserved. CONFIDENTIAL 48