Navigating the Data Landscape: From Fundamentals to the Future

This session unveils the power of data and its transformative impact on organisations. Join us as we explore key topics, including data-intensive application development, reliable and scalable architectures, diverse data models, query languages, storage and retrieval strategies, replication methods, data pipelines, and future data trends.

Discover why data is crucial for driving business success and innovation. Gain insights into building reliable and scalable applications capable of handling vast amounts of data. Explore different data models such as relational, document, and graph, and learn about their use cases. Delve into query languages and storage options, including in-memory databases, data warehouses, data lakes and lake houses.

Understand data flow models via databases, networks (REST, RPC), and async message passing (Kafka, RabbitMQ). Explore replication techniques, data partitioning strategies, and challenges with transactions in distributed systems. Uncover the significance of data pipelines, batch processing, stream processing, ETL, ELT, and reverse ETL. Look at the ideas behind data platforms and see how different technologies are combined to create strong data ecosystems.

Discover how to leverage your knowledge to create a decentralised and scalable data ecosystem—a data mesh. Issues related to distributed systems, consistency, consensus and data governance will be addressed, shedding light on the challenges and considerations involved. Explore the idea of “Data as a product” with essential aspects of managing and deriving value from data assets. Finally, gain insights into emerging data trends and the future of data applications.

Join us in this session to unlock the potential of data, acquire valuable techniques, and prepare for the future of data-driven innovation.

Transcript

1. NAVIGATING THE DATA LANDSCAPE From Fundamentals to the Future
2. CONTENTS WHAT IS DATA DATA MODELS DATA STORAGE and it's importance and query languages and retreival of data DATA REPLICATION and data sharding
3. CONTENTS DATA PLATFORM DATA MESH and it's use cases and distributed systems DATA AS A PRODUCT and future of data and data sharding DATA PIPELINE
4. HI EVERYONE
5. ABOUT ME My name is Mann and I am not a Data Scientist Steve Mann Delhi JUG Leader Senior Software Engineer at Fynd
6. WHAT THIS TALK IS NOT around building data applications a course on big data around understanding forms of data an overview of data tools
7. WHAT IS DATA
9. Data is a magical unicorn made up of ones and zeroes, roaming through cyberspace, collecting cat videos and conspiracy theories along the way WHAT IS DATA
10. WHY IS DATA IMPORTANT Save Time Make Informed Decisions Personalisation and Customer Understanding Risk Management Foster Innovation Transparency and Accountability
11. INTRODUCING Data. Fitness. You
12. BUILDING DATA INTENSIVE APPLICATIONS Reliable Scalable Maintainable
13. RELATIONAL DOCUMENT GRAPH TIME SERIES DATA MODELS
14. user_id name email phone workout_id exercise_id date duration user_id exercise_id name difficulty_level plan_id name exercises price Workout Table User Table Exercise Table Fitness Plan Table RELATIONAL MODEL
15. Workout Document Fitness Plan Document User Document DOCUMENT MODEL
16. User 1 Workout Post 1 User 3 User 2 follows creates User 4 comments on comments on Workout Post 2 creates comments on GRAPH MODEL
17. tags: workout_id=1, user_id=101, exercise_id=201, fields: date=2023-07-01, duration=45 minutes tags: workout_id=2, user_id=102, exercise_id=202, fields: date=2023-07-02, duration=60 minutes TIME SERIES MODEL Workout Measurement tags: exercise_id=201, fields: name=Push-ups, difficulty_level=Intermediate tags: exercise_id=202, fields: name=Squats, difficulty_level=Beginner Exercise Measurement
18. HOW TO QUERY YOUR DATA? SELECT * FROM query_languages WHERE type = 'SQL';
19. QUERY STYLES MAP REDUCE? DECLARATIVE STYLE imperative or declarative? SQL AGGREGATION PIPELINE the javascript SQL IMPERATIVE STYLE YYDT
20. IMPERATIVE STYLE You Yourself Do It YYDT - Pronounced at YUKK 🤢
21. DECLARATIVE STYLE
22. MAP REDUCE QUERYING
23. AGGREGATION PIPELINE
24. DATABASES IN-MEMORY DATA WAREHOUSES DATA LAKES OBJECT STORAGE WHERE TO STORE DATA?
25. JSON XML AVRO PARQUET CSV DATA ENCODING
26. VIA DATABASE VIA REST (OR SOAP) VIA ASYNC MESSAGING PASSING DATA BETWEEN SERVICES?
27. VIA DATABASE Simple to implement Needs to be backward compatible Needs to be forward compatible
28. Old version of code (missing description) DATA OUTLIVES CODE Data written by new code Description is lost
29. VIA REST Easier to develop Flexible (loose coupling) Popular (community support) MICRO-SERVICES!
30. Server Database Client RESTful API over HTTP Who needs a nap when you have REST to keep you RESTed
31. VIA ASYNC MESSAGING Event Driven Architecture Fault Tolerant Asynchronous Flow MICRO-SERVICES!
32. Server Database Client Message Huge shoutout to the clients Broker Message Client Message
33. WE CAN HANDLE DATA!
34. BUT...
35. SHARDING Split a single dataset into partitions or shards All shards run on separate nodes Leverage horizontal scaling Increased throughput
36. user_id name age 1 Venkat 31 2 Josh 28 3 Ivar 30 4 Mala 23 Sharding User Table SHARDING user_id name age 1 Venkat 31 2 Josh 28 user_id name age 3 Ivar 30 4 Mala 23 Sharding
37. DATA REPLICATION Increased throughput Scalability Fault tolerance
38. User Read-Write Query Leader Data change Data Change Replication Streams Follower Follower LEADERS AND FOLLOWERS
39. TIME TO DIFFERENTIATE BETWEEN OLTP AND OLAP
40. DATA PIPELINES Flexibility Scalability Separation of Concern
41. extract extract extract extract load load load load EXTRACT-TRANSFORM-LOAD (ETL) Workout DB Nutrition DB User DB Wearable Sales DB OLTP Systems Data Warehouse Transform Transform Transform Transform OLAP System
42. E & L extract and load E & L extract and load EXTRACT-LOAD-TRANSFORM (ELT) Workout DB Nutrition DB User DB Wearable Sales DB OLTP Systems Data Warehouse / Lake OLAP System Transform
43. DATA PLATFORM Application Layer Security Layer Authentication, Authorisation, Logging, Alerting Web Apps, Microservices, Enterprise Applications Storage Layer OLTP Systems and Databases Ingestion Layer APIs, ETL, ELT, Pub Sub, Data Streams Analytics Layer OLAP Systems, Data Warehouses, Data Lakes Data Governance
44. DATA GOVERNANCE Security Access and Availability Quality Compliance
45. ISSUES WITH DISTRIBUTED SYSTEMS System Faults and Partial Failures Unreliable Networks Unreliable System Clocks Questionable Reality
46. FUTURE TRENDS
47. load load load load extract extract extract extract REVERSE ETL Workout DB Nutrition DB User DB Wearable Sales DB OLTP Systems Data Warehouse / Lake Transform Transform Transform Transform OLAP System
48. DATA MESH Data Infrastructure as a Platform Domain 1 Domain 3 Domain 2 Domain 4
49. MESH OR NOT TO MESH More about process than architecture Domain Centric Segregation Federated Governance Data as a product
50. DATA AS A PRODUCT DaaP is a mindset Treats data as a valuable asset Data Ownership Introduces data interfaces
51. FEEL FREE TO REACH OUT
52. THANK YOU!