Wins in Data Engineering

Transcript

1. Juhi Singh Hidden Wins in Data Engineering
2. I’m Juhi Singh Data Engineer, Coursera jsingh@coursera.org LinkedIn @singhjuhi91 medium.com/coursera-engineering Hi there!
3. We envision a world where anyone, anywhere can transform their life by accessing the world’s best learning experience.
4. Deep Learning
5. Deep Learning Machine Learning
6. Deep Learning Machine Learning ML with TensorFlow on Google Cloud
7. Our Impact at Scale 35 million+ Learners 120 million+ Enrollments 3000+ Courses 200+ Content Partners
8. Our Data at Scale 200 Million Events/day 100 GB Writes/Day 200 TB Reads/Day 30 TB In Database
9. Why is this important?
10. Maslow’s Hierarchy of Needs Link
11. The AI Hierarchy of Needs Link
12. Teams in Data Science
13. The AI Hierarchy of Needs Data Engineers Data Scientists Data Infra Engineers Teams in Data Science
14. The AI Hierarchy of Needs Curating Capitalizing Collecting Three Cs in Data Science
15. 15 User Activity Collect Curate Capitalize Extract, Load Transform In-Platform Dashboards User Recommendations Research Exports Coursera Data Flow Third Party Tools Event Feeds Amazon S3 Cassandra DW - Redshift
16. 16 User Activity Collect Curate Capitalize Transform In-Platform Dashboards User Recommendations Research Exports Coursera Data Flow Third Party Tools Event Feeds Amazon S3 Cassandra DW - Redshift DW - Redshift Extract, Load
17. 17 User Activity Collect Curate Capitalize Transform In-Platform Dashboards User Recommendations Research Exports Coursera Data Flow Third Party Tools Event Feeds Amazon S3 Cassandra DW - Redshift DW - Redshift Extract, Load
18. 18 Collect Curate Capitalize Transform In-Platform Dashboards User Recommendations Research Exports Coursera Data Flow DW - Redshift DW - Redshift
19. 19 Collect Curate Capitalize In-Platform Dashboards User Recommendations Research Exports Coursera Data Flow DW - Redshift
20. 20 User Activity Collect Curate Capitalize Transform In-Platform Dashboards User Recommendations Research Exports Coursera Data Flow Third Party Tools Event Feeds Amazon S3 Cassandra DW - Redshift DW - Redshift Extract, Load
21. Hidden Wins of Data Engineering Link
22. 1. There should be one, and preferably only one obvious way to do it
23. 23 User Activity Collect Curate Capitalize Transform In-Platform Dashboards User Recommendations Research Exports One and Only One Third Party Tools Event Feeds Amazon S3 Cassandra DW - Redshift DW - Redshift Extract, Load
24. 24 Collect Curate Capitalize Transform In-Platform Dashboards User Recommendations Research Exports Coursera Data Flow DW - Redshift DW - Redshift
25. Schema Hierarchy
26. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events Table-2 Table-3 Table-4 Table-5 Table-8 Table-9 Table-11 Table-1 Table-12 Table-13 Table-14 Table-15 Table-6 Table-7 Level-0 Level-1 Level-2 Level-3 Table-10 Schema Hierarchy
27. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events Table-2 Table-3 Table-4 Table-5 Table-8 Table-9 Table-11 Table-1 Table-12 Table-13 Table-14 Table-15 Table-6 Table-7 staging prod core reporting Table-10 Schema Hierarchy
28. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events staging.s3_degrees staging.gsheet_deadlines staging.mega_users staging staging.* schema - Get data into EDW - Table name implied source - staging.mysql_* - staging.s3_* - No PK, FK constraints - Managed by - Product Engineers - Data Infra Engineers - Data Engineers - Data Scientists Schema Hierarchy
29. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events Table-2 Table-3 prod.users prod.courses prod.course_domains Table-1 prod.transactions prod.course_branches staging prod prod.* schema - Sensible Data - 3NF - PK, FK constrained - Globally unique column names - Strict naming conventions - Managed by - Data Engineers - Product Engineers Schema Hierarchy
30. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events Table-2 Table-3 Table-4 Table-5 Table-8 core.courses Table-1 Table-6 Table-7 staging prod core core.course_enrollments core.* schema - Standardized - Aggregated - Denormalized* - PK constrained - Documented - Grain-specific - User Friendly - Managed by - Data Engineers Schema Hierarchy core.user_course_items
31. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum reporting.daily_enrollments reporting.recruitment_metrics reporting.app_leads reporting.course_drops reporting reporting.* schema - External - Super-fat - Super-efficient - “No Join” - Use-case specific - Only powered through core - Managed by - Data Engineers Schema Hierarchy
32. There should be one, and only one obvious way to do it One obvious way of Transforming your data Structuring your data Labeling your data
33. 2. Errors should never pass silently
34. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events Table-2 Table-3 Table-4 Table-5 Table-8 Table-9 Table-11 Table-1 Table-12 Table-13 Table-14 Table-15 Table-6 Table-7 6:00 AM 7:00 AM 8:00 AM 9:00 AM Table-10 staging prod core reporting Scheduled Workflow
35. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events Table-2 Table-3 Table-4 Table-5 Table-8 Table-9 Table-11 Table-1 Table-12 Table-13 Table-14 Table-15 Table-6 Table-7 6:00 AM 7:00 AM 8:00 AM 9:00 AM Table-10 Scheduled Workflow
36. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events Table-2 Table-3 Table-4 Table-5 Table-8 Table-9 Table-11 Table-1 Table-12 Table-13 Table-14 Table-15 Table-6 Table-7 6:00 AM 7:00 AM 8:00 AM 9:00 AM Table-10 Scheduled Workflow
37. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events Table-2 Table-3 Table-4 Table-5 Table-8 Table-9 Table-11 Table-1 Table-12 Table-13 Table-14 Table-15 Table-6 Table-7 6:00 AM 7:00 AM 8:00 AM 9:00 AM Table-10 Scheduled Workflow ?
38. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events Table-2 Table-3 Table-4 Table-5 Table-8 Table-9 Table-11 Table-1 Table-12 Table-13 Table-14 Table-15 Table-6 Table-7 6:00 AM 8:05 AM Table-10 staging prod core reporting User Events Dependent Workflow
39. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events Table-2 Table-3 Table-4 Table-5 Table-8 Table-9 Table-11 Table-1 Table-12 Table-13 Table-14 Table-15 Table-6 Table-7 6:00 AM Table-10 Dependent Workflow
40. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events Table-2 Table-3 Table-4 Table-5 Table-8 Table-9 Table-11 Table-1 Table-12 Table-13 Table-14 Table-15 Table-6 Table-7 6:00 AM Table-10 Dependent Workflow
41. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events Table-2 Table-3 Table-4 Table-5 Table-8 Table-9 Table-11 Table-1 Table-12 Table-13 Table-14 Table-15 Table-6 Table-7 6:00 AM Table-10 Dependent Workflow
42. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events Table-2 Table-3 Table-4 Table-5 Table-8 Table-9 Table-11 Table-1 Table-12 Table-13 Table-14 Table-15 Table-6 Table-7 6:00 AM Table-10 Dependent Workflow
43. Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum User Events Table-2 Table-3 Table-4 Table-5 Table-8 Table-9 Table-11 Table-1 Table-12 Table-13 Table-14 Table-15 Table-6 Table-7 6:00 AM Table-10 Dependent Workflow
44. 1. Root Jobs a. Jobs without upstream dependencies b. Invoked at certain scheduled time Workflow Management Scheduled Jobs a. Jobs with upstream dependencies b. Invoked only if upstream have finished running
45. 1. Root Jobs a. Jobs without upstream dependencies b. Invoked at certain scheduled time Workflow Management 2. Dependent Jobs a. Jobs with upstream dependencies b. Invoked only if upstream have finished running
46. 1. You will “own” a table if you a. Create a table b. Modify a table c. Schedule its job 2. Notified when a. Health Check Fails b. Dropped c. Stale d. Not documented Table Ownerships
47. 1. Integrated with a. Slack b. Email c. Database Alerting
48. Errors should never pass silently Workflow Management is your friend Assign Table Owners Automate Table Health Checks Integrate Notifications
49. Coursera Data Flow MEGA
50. Coursera Data Flow MEGA
51. MEGA
52. MEGA - Make EDW Great Again MEGA
53. MEGA
54. MEGA
55. MEGA
56. 1. There should be only ???, and preferably only ??? way of doing it Fill in the Blanks!
57. Fill in the Blanks! 1. There should be only one, and preferably only one way of doing it Standardize Schema Hierarchy → Transform your Data Enforce Schema Modeling Principles → Structure your Data Field Naming Conventions → Label your Data Fill in the Blanks!
58. 2. ??? should never pass silently Fill in the Blanks! Fill in the Blanks!
59. 2. Errors should never pass silently Workflow Management Assign Table Owners Automate Table Health Checks Integrate Notifications Fill in the Blanks! Fill in the Blanks!
60. Coursera Data Flow How to Build a Car
61. Juhi Singh Data Engineer, Coursera jsingh@coursera.org LinkedIn @singhjuhi91 medium.com/coursera-engineering Thank You!
62. Q & A