What Is Data Engineering

A series of talks on data engineering

Transcript

1. DATA ENGINEERING DATA ENGINEERING ESSENTIALS ESSENTIALS 1
2. PLAN PLAN pillars of data engineering bird's eye view on ecosystem and typical architectures building blocks of big data and what's in common real-world pipeline example 2 . 1
3. WHO WE ARE AND WHERE WE ARE WHO WE ARE AND WHERE WE ARE we are data engineers building data pipelines using building blocks vocab: data pipeline 3 . 1
4. RECAP: DATA PIPELINE RECAP: DATA PIPELINE 3 . 2
5. 3 . 3
6. data ows into the system from various data sources and has to be prepared for serving what is the trigger? incoming data (push/pull) 3 . 4
7. 3 . 5
8. TYPES OF WORKLOAD TYPES OF WORKLOAD Ingestion and storage of the big datasets Batch processing of data at rest Real-time streaming processing of data in motion Interactive exploration of big data Predictive analytics and machine learning vocab: job ETL ingestion, streaming, batch data-at-rest, data-at-motion data wrangling, data mining 3 . 6
9. BUILDING BLOCKS BUILDING BLOCKS
10. 3 . 7
11. how do they work? what's in common? 3 . 8
12. PILLARS OF DATA ENGINEERING PILLARS OF DATA ENGINEERING 4 . 1
13. RECAP: 3VS RECAP: 3VS Volume Velocity Variaty 4 . 2
14. DISTRIBUTING: DATA, COMPUTATION, ACCESS DISTRIBUTING: DATA, COMPUTATION, ACCESS distributed storage distributed computation distributed access 4 . 3
15. DISTRIBUTED SYSTEMS DISTRIBUTED SYSTEMS 4 . 4
16. TYPES OF PARALLELISM TYPES OF PARALLELISM data parallelism task parallelism 4 . 5
17. 4 . 6
18. PARTITIONING PARTITIONING query patterns materialized viewes vocab: partitioning 4 . 7
19. REPLICATION REPLICATION 4 . 8
20. READ-WRITE SEMANTICS READ-WRITE SEMANTICS 4 . 9
21. CAP THEOREM CAP THEOREM vocab: CAP 4 . 10
22. COMMON BIG-DATA-SYSTEM ARCHITECTURE COMMON BIG-DATA-SYSTEM ARCHITECTURE (building block: framework / tool) nodes (workers/slaves/brokers/nodes/…) memory, cpu, storage + client + coordinator + master coordinator scheduler metastorage 4 . 11
23. WHAT IS APPLICATION THEN? WHAT IS APPLICATION THEN? 4 . 12
24. DISTRIBUTED ALGORITHMS & COORDINATION DISTRIBUTED ALGORITHMS & COORDINATION leader election consensus non-blocking data structures replication atomic broadcast consistent hashing .. 4 . 13
25. LAWS OF DISTRIBUTED COMPUTING LAWS OF DISTRIBUTED COMPUTING 4 . 14
26. MOORE MOORE
27. 4 . 15
28. AMDAHL AMDAHL 4 . 16
29. COMMON BIG DATA TOOLS & FRAMEWORKS COMMON BIG DATA TOOLS & FRAMEWORKS how do they work? how to get our heads over this in fact this is all about microservices 5 . 1
30. ECOSYSTEM VIEW ECOSYSTEM VIEW
31. 5 . 2
32. WHAT'S IN COMMON WHAT'S IN COMMON storage, data at-rest hdfs (AP, distributed le system) hive, presto, tez, impala (sql engine) bigtable/hbase (CP, random-access, OLTP) dynamo, cassandra (AP) elastic (FTS, AP) … processing, data at-motion mapreduce (classic programming model) spark (in-memory processing) ink, storm, beam kafka (streaming, messaging) … orchestration, mngmnt, work ow zookeeper (coordination) air ow, oozie (work ow) yarn (cluster management) … 5 . 3
33. GUIDELINE TO GET INTO NEW BIG DATA FRAMEWORK GUIDELINE TO GET INTO NEW BIG DATA FRAMEWORK at glance storage / computation? where is it in terms of CAP? olap / oltp, read or write optimized? stream or batch? data model computation model delivery semantics what kind of interface it provides? (client / con guration) 5 . 4
34. GUIDELINE TO GET INTO NEW BIG DATA FRAMEWORK GUIDELINE TO GET INTO NEW BIG DATA FRAMEWORK deep what services it runs? where it runs those? what do they do? how they communicate? 5 . 5
35. TYPICAL SOLUTION ARCHITECTURES TYPICAL SOLUTION ARCHITECTURES 6 . 1
36. LAMBDA LAMBDA 6 . 2
37. KAPPA KAPPA 6 . 3
38. DATA LAKE DATA LAKE 6 . 4
39. vocab: structured & unstructured data schema-on-read data warehouse OLAP / OLTP 6 . 5
40. REAL-WORLD PIPELINE REAL-WORLD PIPELINE 7 . 1
41. 7 . 2
42. WHAT'S NEXT? WHAT'S NEXT? getting hands dirty word-count problem literature designing data-intensive applications courses ask me let's meet next week :) https://www.coursera.org/learn/big-data- analysis 8 . 1
43. 😃 😃 9 . 1