and where is it going? 1. How strides in technology enabled the current state of data platforms 2. The essential components & features that make up a modern data stack 3.What data teams need to know to take advantage of emerging trends
to be data-driven, teams are using 100s of tools to do so. The most important difference between a modern data stack and a legacy data stack is that the modern data stack is hosted in the cloud, requires little technical configuration, and opens up the barrier to entry for more users.
and strengthened their features, innovation is king. The History of the MDS Pre-2012 Business data scattered across siloed databases, machine-generated data relied on Hadoop, analytics lived in local Excel spreadsheets. 6 2012 Amazon Redshift is released - this is regarded as the historical event that caused a major shift. 2022 The modern data stack is becoming and setting the standard in data & analytics.
executives Cloud-native destinations with limited support for processing to transform data Customized scripts tailored to each data source to extract and transform data ETL is standard. More data lives in more places — and needs to get it to other places, including now the cloud.
Many Use Cases Huge Audience Modern Data Stack Welcome, ELT. Rapid expansion of tools creating data, destination types, and departments to serve mean data needs to be more accessible by more people.
Schema n table 1 table n table 1 table n Marketing Analytics Databases Files Finance Analytics Product Analytics Sales Analytics Data Sources Data Pipeline Replicated Data Raw Database Schema 1 Schema n table 1 table n table 1 table n Transformation Inputs Transformation Outputs Cloud Data Warehouse Business Intelligence Data Transformations Reverse ETL Embedded Analytics Ad Hoc Reporting Data Science & AI/ML Data Lake Data Sources Ingestion Destinations Transformations Output Event Collectors
of platform that stores your data. Common classifications for 1st party data include: • Databases • Files • Applications, categorized by use case ◦ i.e. marketing analytics, finance analytics • Event Collectors 12 Data Sources Marketing Analytics Databases Files Finance Analytics Product Analytics Sales Analytics Data Sources Event Collectors
sources and loads that data to your destination. A modern data pipeline needs to be: • Low code / no code • Normalized & automated • Maintained for data integrity 13 Ingestion Data Pipeline Replicated Data
lands. Destinations in the Modern Data Stack are natively cloud-based, compute and store more data at lower costs. • Data warehouses are relational databases designed to store and transform data, best for quickly analyzing historical data. • Data lakes hold high volumes of data in a raw format, supporting structured, unstructured, and semi-structured types of data, best for building AI models. 14 Destinations Modeled Data Schema 1 Schema n table 1 table n table 1 table n Raw Database Schema 1 Schema n table 1 table n table 1 table n Transformation Inputs Transformation Outputs Cloud Data Warehouse Data Transformations Data Lake
computing, separating and combining raw data into analysis-ready data models. A modern transformations tool needs to be: • Automated and regular • Ensure data integrity • In SQL, common language for analysts and engineers 15 Transformations Modeled Data Schema 1 Schema n table 1 table n table 1 table n Raw Database Schema 1 Schema n table 1 table n table 1 table n Transformation Inputs Transformation Outputs Cloud Data Warehouse Data Transformations Data Lake
be put to use. Where data goes to get analyzed. Needs to be accessible, easy to understand. • Business intelligence • Embedded analytics • Ad hoc reporting Newer to the modern data stack. • Reverse ETL: Process of loading data from a data warehouse into an application or tool • Data Science & AI/ML: Enables machines to make decisions based on learned data models Modern Data Stack 16 Output Business Intelligence Reverse ETL Embedded Analytics Ad Hoc Reporting Data Science & AI/ML
expensive to scale - cloud destinations enable higher volumes of data movement & compute at a lower cost. 18 With cloud-first technology growth, ETL evolved into ELT, enabling post-load transformations execution in the cloud destination. SQL is inherent in the MDS for data modeling which allows data analysts to action earlier in the MDS, in transformations. Data as an automated service is the cornerstone of the MDS, instead of being created and maintained by you & your team. ETL → ELT SQL-Based Fully Managed Essential Features
significantly faster rate & reduced cost Higher Volumes + Lower Costs Data Accessibility Self-service analytics programs increase your organization’s data literacy Built to Scale Pricing & products supports scale for data volumes, # of users, and use cases Best-in-Breed Technologies Specialization drives innovation and modularity gives your team flexibility The Modern Data Stack: The Benefits
data connectors • Automatic schema migration • Integrated transformations Target Destinations: • Cloud Data Warehouses • Cloud Data Lakes Support for Downstream Use Cases: • Data-driven models (Customer Journey or Experience) • BI reporting / visualizations • Powering customer applications Data Sources Automated Ingestion Cloud Infrastructure Transformation Output Data Science & AI /ML (Domino, DataRobot, DataIku, Sagemaker…) Operational Sync (Census, Thoughtspot)) Ad Hoc (Tableau, Google Sheets, Excel) Dashboards (Looker, Tableau, Superset, Mode) Embedded Analytics (Sisense, Looker) Augmented Analytics (Thoughtspot, Sisu) OLTP Databases via CDC Applications/ERP (Oracle, Salesforce, Netsuite…) Event Collectors (Segment, Snowplow) Integrated Transform s Normalized Schemas Logs 3rd Party APIs (Zuora, Stripe, Facebook) File and Object Storage Data Lake Fivetran Data Flows Other Data Flows Dimensional Schemas Cloud Data Warehouse Managed Data Connectors
secure data movement. Your network (Cloud) Data Lake Source Agent* Data Warehouse Event Stream (Kafka) File Your network (Cloud / On-premise) Events Application Databases Files Destination Agent *High Volume Real Time Replication Fivetran Reference Architecture HVR Hub Unpack/ Appconnect/Table Explore
whether it's via SSH, VPN, or PrivateLink. Anonymize personal data Block or hash sensitive data before it touches a destination. Data doesn’t persist Data is purged as soon as it’s successfully written to the destination. Transparency & auditability Full control over data access and detailed logging means full control over how data is handled — and by whom. Control data access Granular role-based access control ensures users get the right level of access. Control user access Role based access control and user provisioning via SSO providers. CCPA, GDPR, HIPAA, ISO, PCI, SOC2
is evolving into the Modern Data Platform. Now, concept of “lakehouse” creates convergence of traditional BI and AI/ML/Multimodel data processing infrastructures. Fivetran is positioned to the key data movement solution for the MDP (ingestion & transformation). 28
mark3t1ng
jcasabona
tmiket
bryan
3n
tanoku
eileencodes
katiecoart
jonrohan
tammyeverts
seathinner
notwaldorf
szymonslowik
