What Is Big Data

A series of talks on data engineering

Transcript

1. WHAT IS BIG DATA? WHAT IS BIG DATA? AND WHY YOU MAY CARE AND WHY YOU MAY CARE 1
2. PLAN PLAN What is BigData? Different perspectives Use-cases 2 . 1
3. 1 1 MESSAGE MESSAGE informative motivational 3 . 1
4. 2 2 WHY IT MATTERS TO YOU WHY IT MATTERS TO YOU
5. more and more data investment into your career 4 . 1
6. 3 3 A FEW CASES A FEW CASES 1Tb per day at stock exchange 500+Tb at facebook each day 10+Tb in 30mins of Jet ight solar station can generate more data then network to transmit to cloud 73% of data is not used these days 5 . 1
7. 4 4 WHAT IS (BIG)DATA? WHAT IS (BIG)DATA? cumulative term huge amount of data business problems related to collecting and retrieving value from the (big) data a set of engineering tools, patterns, programming models and skills to solve those problems 6 . 1
8. 3Vs Volume Velocity Variaty 6 . 2
9. 6 . 3
10. 5 5 BUSINESS: WHY IT MATTERS BUSINESS: WHY IT MATTERS data driven decisions operational ef ciency, risk management, etc. cost optimization .. 7 . 1
11. 6 6 BUSINESS: LEVELS OF ADOPTION & BUSINESS: LEVELS OF ADOPTION & CUSTOMER PAIN CUSTOMER PAIN A given customer can be at various levels of big-data / AI adoption, which re ects his current needs, problems and pain points 8 . 1
12. 8 . 2
13. 1. early: "Why my MySQL instance can't handle this?" 8 . 2
14. 1. early: "Why my MySQL instance can't handle this?" 2. optimization & strategy required: "shi.. my aws bill" 8 . 2
15. 1. early: "Why my MySQL instance can't handle this?" 2. optimization & strategy required: "shi.. my aws bill" 3. "I have money, let's build something" - lack of strategy: operational, technical 8 . 2
16. 1. early: "Why my MySQL instance can't handle this?" 2. optimization & strategy required: "shi.. my aws bill" 3. "I have money, let's build something" - lack of strategy: operational, technical 4. data swamp: "where are my socks?" 8 . 2
17. 1. early: "Why my MySQL instance can't handle this?" 2. optimization & strategy required: "shi.. my aws bill" 3. "I have money, let's build something" - lack of strategy: operational, technical 4. data swamp: "where are my socks?" 5. platform already exists: Analytics, BI, ML 8 . 2
18. 1. early: "Why my MySQL instance can't handle this?" 2. optimization & strategy required: "shi.. my aws bill" 3. "I have money, let's build something" - lack of strategy: operational, technical 4. data swamp: "where are my socks?" 5. platform already exists: Analytics, BI, ML 6. All of them: "Here I planted my potatoes.. want to adopt AI/ML eventually.." 8 . 2
19. 7 7 BIGDATA SOLUTION BIGDATA SOLUTION 9 . 1
20. 9 . 2
21. 9 . 3
22. ENGINEERING ENGINEERING distributed systems structured, unstructured data collect (ingest, integrate) prepare: clean, enrich, feature extract, pre- aggregate, .. process batch real-time store integrate (user interface - SQL front-end engines, API REST front-ends, services) 9 . 4
23. 9 . 5
24. BI BI analytics, insights query/reason about vizualise reporting 9 . 6
25. 9 . 7
26. ML/AI ML/AI explore predict recommend 9 . 8
27. 9 . 9
28. GOVERNANCE GOVERNANCE inventory metadata management data quality security governing .. 9 . 10
29. 8 8 DELIVERY & TEAM MANAGEMENT DELIVERY & TEAM MANAGEMENT PERSPECTIVE PERSPECTIVE 10 . 1
30. 10 . 2
31. agile doesn't (fully) work 10 . 2
32. agile doesn't (fully) work data pipeline is bad to split vertically 10 . 2
33. agile doesn't (fully) work data pipeline is bad to split vertically POC phase & not-ending exploration 10 . 2
34. agile doesn't (fully) work data pipeline is bad to split vertically POC phase & not-ending exploration team: smaller over versatile 10 . 2
35. agile doesn't (fully) work data pipeline is bad to split vertically POC phase & not-ending exploration team: smaller over versatile requirements collection –> data exploration 10 . 2
36. agile doesn't (fully) work data pipeline is bad to split vertically POC phase & not-ending exploration team: smaller over versatile requirements collection –> data exploration BA –> BI 10 . 2
37. agile doesn't (fully) work data pipeline is bad to split vertically POC phase & not-ending exploration team: smaller over versatile requirements collection –> data exploration BA –> BI QA –> Data Engineer 10 . 2
38. agile doesn't (fully) work data pipeline is bad to split vertically POC phase & not-ending exploration team: smaller over versatile requirements collection –> data exploration BA –> BI QA –> Data Engineer DevOps –> DataOps Engineer 10 . 2
39. agile doesn't (fully) work data pipeline is bad to split vertically POC phase & not-ending exploration team: smaller over versatile requirements collection –> data exploration BA –> BI QA –> Data Engineer DevOps –> DataOps Engineer tooling matters A LOT: dev environment, fast access, .. 10 . 2
40. agile doesn't (fully) work data pipeline is bad to split vertically POC phase & not-ending exploration team: smaller over versatile requirements collection –> data exploration BA –> BI QA –> Data Engineer DevOps –> DataOps Engineer tooling matters A LOT: dev environment, fast access, .. simplicity of architecture over structure 10 . 2
41. agile doesn't (fully) work data pipeline is bad to split vertically POC phase & not-ending exploration team: smaller over versatile requirements collection –> data exploration BA –> BI QA –> Data Engineer DevOps –> DataOps Engineer tooling matters A LOT: dev environment, fast access, .. simplicity of architecture over structure optimization over simplicity 10 . 2
42. agile doesn't (fully) work data pipeline is bad to split vertically POC phase & not-ending exploration team: smaller over versatile requirements collection –> data exploration BA –> BI QA –> Data Engineer DevOps –> DataOps Engineer tooling matters A LOT: dev environment, fast access, .. simplicity of architecture over structure optimization over simplicity sacri ce locally to optimize globally 10 . 2
43. agile doesn't (fully) work data pipeline is bad to split vertically POC phase & not-ending exploration team: smaller over versatile requirements collection –> data exploration BA –> BI QA –> Data Engineer DevOps –> DataOps Engineer tooling matters A LOT: dev environment, fast access, .. simplicity of architecture over structure optimization over simplicity sacri ce locally to optimize globally communicate: half of something is better then all of nothing 10 . 2
44. 9 9 ARCHITECTURE DESIGN ARCHITECTURE DESIGN 11 . 1
45. batch streaming 11 . 2
46. non-functional requirements - crucial latency throughput cost (infrastructure, development, support(complexity)) .. 11 . 3
47. typical architectures 11 . 4
48. lambda 11 . 5
49. kappa 11 . 6
50. data lake 11 . 7
51. 10 10 ENGINEERING ENGINEERING 12 . 1
52. WHAT MY MOM THINKS I DO.. WHAT MY MOM THINKS I DO.. 13 . 1
53. skills 30% coding 25% algorithms, distributed design 20% devops 15% sql and statistics 10% visualization & presentation 13 . 2
54. activities 30% debug 30% proof of concept 20% data exploration 15% learn 5% implementation 13 . 3
55. 11 11 HISTORICAL PERSPECTIVE HISTORICAL PERSPECTIVE Pre: Oracle, Informatica, mysql,.. 2004: Hadoop, mapreduce, hdfs 2009-2010: Spark, Kafka Con uent, Databricks, Cloudera, Hortonworks, MapR AWS, Azure, GCP, IBM 14 . 1
56. 14 . 2
57. 12 12 LANDSCAPE LANDSCAPE 15 . 1
58. 15 . 2
59. 13 13 GLOSSARY GLOSSARY BA vs BA vs BI Data Engineer Data Warehouse Data Pipeline Data Lake OLAP, OLTP, OLAP cubes, tensors batch vs stream 16 . 1
60. Data Science vs AI vs ML predictive analytics MPP Hadoop MapReduce, HDFS structured / unstructured data ETL 16 . 2
61. Ingestion Columnar storage Data Governance Data Mining Data Wrangling 16 . 3
62. 14 14 WHAT'S NEXT? WHAT'S NEXT? 17 . 1
63. 😃 😃 18 . 1