Share
Embed
Transcript
49 slides
-
F R O M Z E R O T O
L A K E H O U S E FIRST STEPS WITH DATABRICKS Paul Karikari Principal Solutions Architect- Data & AI, TheDataLead AI 27 June, 2025 TheDataLead AI -
W E L C O M E & A G
E N D A Introduction Data Management The Library Analogy Lakehouse Platform Overview Hands-On Demo Q&A + Wrap-Up T A B L E O F C O N T E N T S TheDataLead AI -
"Without data, you're just another person with a Wi-Fi connection
and a strong opinion." 😄 TheDataLead AI -
Why do businesses NEED data? TheDataLead AI Because "gut instinct"
doesn’t scale past five people and a Slack channel. Because guessing your customer’s needs is so 1995. Helping businesses figure out what’s actually happening… Instead of just arguing about it in long meetings. -
W H Y D O B U S I N
E S S E S N E E D D A T A ? TheDataLead AI 1. Understanding Customer Behaviour Track preferences, behaviour, and feedback. Example: An e-commerce company uses purchase history to recommend products. -
TheDataLead AI 2. Improving Products and Services Identify what works
and what doesn’t. Use data from usage patterns, error reports, and surveys to improve features. W H Y D O B U S I N E S S E S N E E D D A T A ? -
TheDataLead AI 3. Better Decision-Making Replace gut-feeling with evidence-based strategies.
Data helps forecast trends, plan budgets, and choose where to invest. W H Y D O B U S I N E S S E S N E E D D A T A ? -
TheDataLead AI 4. Optimising Operations Improve efficiency in supply chains,
staffing, or manufacturing. Example: Analysing delivery routes to reduce shipping time and cost. W H Y D O B U S I N E S S E S N E E D D A T A ? -
TheDataLead AI 5. Staying Competitive Benchmark against industry data. Spot
opportunities and threats earlier than rivals. W H Y D O B U S I N E S S E S N E E D D A T A ? -
TheDataLead AI 6. Compliance and Reporting Businesses need data to
meet legal, tax, and regulatory requirements. Example: Financial records, GDPR audit trails. W H Y D O B U S I N E S S E S N E E D D A T A ? -
TheDataLead AI 7. Driving Innovation Power machine learning, personalisation, and
automation. Data fuels new business models and smarter experiences (e.g., Netflix recommendations, Tesla's self-driving algorithms). W H Y D O B U S I N E S S E S N E E D D A T A ? -
T H E D A T A C H A
L L E N G E TheDataLead AI Businesses are flooded with data but struggle to turn it into decisions. One team wants raw data fast. Another needs trusted, curated insights. Juggling costs, access, and governance. -
E V O L U T I O N O
F D A T A S T O R A G E S Y S T E M S TheDataLead AI D I S K S T O R A G E R E L A T I O N A L D B M S D A TA WA R E H O U S E D A TA L A K E L A K E H O U S E -
OLTP: Like a cash register – fast, accurate, handles lots
of small transactions. OLAP: Like a reporting dashboard – tells you items sold, when, and why. TheDataLead AI -
Why did OLTP break up with OLAP? Because every time
OLTP wanted to keep things quick and transactional, OLAP wanted to “analyze the relationship” from every possible angle. 😅 TheDataLead AI -
Books Catalog Librarians Readers Shelves Governance T H E W
O R L D O F D A T A I S L I K E A L I B R A R Y What If Data Were Like a Library? Just like a library has structure, organisation, and purpose, so should your data. TheDataLead AI Raw data files (CSV, JSON) Metadata tables and schemas Data engineers maintaining order Business users & analysts Storage layers Library Policies -
DATA WAREHOUSE : THE CURATED ROOM T R A D
I T I O N A L D A T A W A R E H O U S E S TheDataLead AI T H E L I B R A R Y A N A L O G Y Once upon a time, knowledge was kept in a physical library. Only clean, pre-approved books could enter. Everything was in order until someone asked for a rough draft or a sketch. The system worked well, but it was inflexible. -
DATA WAREHOUSE : THE CURATED ROOM Curated, cleaned content only
Limited types of books allowed High performance, but rigid T R A D I T I O N A L D A T A W A R E H O U S E S TheDataLead AI T H E L I B R A R Y A N A L O G Y -
TheDataLead AI The polished library had rules. But soon, businesses
needed messy notes, images, recordings data that didn’t fit into tidy rows. They needed instant updates, machine learning experiments, real-time insights. The library couldn’t keep up. T H E L I B R A R Y A N A L O G Y L I M I T A T I O N S O F T R A D I T I O N A L D A T A W A R E H O U S E S WHY THE LIBRARY MODEL BROKE DOWN -
TheDataLead AI Limited types of books allowed - Structured Data
only Slow to update means delayed data availability Not Built for AI/ML because no access to raw, historical or unstructured data. T H E L I B R A R Y A N A L O G Y WHY THE LIBRARY MODEL BROKE DOWN L I M I T A T I O N S O F T R A D I T I O N A L D A T A W A R E H O U S E S -
DATA LAKE : THE BASEMENT ARCHIVE T R A D
I T I O N A L D A T A L A K E S TheDataLead AI In frustration, the city built a giant basement to hold everything. No rules, no structure just piles of books, napkins, and notes. It was cheap to build, but impossible to navigate. Insights were buried under chaos -
DATA LAKE : THE BASEMENT ARCHIVE All data dumped in
raw form Great for storage, not for access No structure, hard to find anything T R A D I T I O N A L D A T A L A K E S TheDataLead AI -
TheDataLead AI DATA MANAGEMENT : WHY IT IS AN ISSUE
Data Lakes vs Data Warehouse -
E N T E R T H E L A
K E H O U S E : A S M A R T , U N I F I E D L I B R A R Y T H E B E S T O F B O T H W O R L D S TheDataLead AI Then came a new kind of library which was open yet organized. It welcomed raw ideas and polished books. Auto-sorting shelves and robotic assistants. Fast, smart, and flexible. This called the Lakehouse -
W H A T I S T H E D
A T A B R I C K S L A K E H O U S E ? O N E U N I F I E D P L A T F O R M Combines the structure of warehouses with the scalability of data lakes Built on Delta Lake (open source) Supports SQL, Python, ML, streaming, and more Perfect for collaboration and governance TheDataLead AI -
M E D A L L I O N A
R C H I T E C T U R E A logical organization of data in a Lakehouse TheDataLead AI -
Component Analogy Role Delta Lake Smart bookshelves ACID storage, versioning
Apache Spark Fast librarians Scans 1000s of books instantly MLflow Data scientist’s notebook Tracks predictions and experiments Unity Catalog Access control panel Who read what, and when K E Y C O M P O N E N T S Your Smart Library Staff TheDataLead AI -
D A T A M O D E L :
L I B R A R Y S Y S T E M TheDataLead AI Structure & Clarity : Organizes data into understandable formats. Business Alignment : Enforces consistent definitions and rules. Reusability : Enables multiple teams to build on clean, reliable data. Scalability : Makes it easy to grow the pipeline or add new features. Governance : Supports compliance, documentation, and data lineage Why a Data Model Matters -
Keeps logs of all readers and access Controls who can
borrow special collections (e.g., sensitive data) Helps with audits and compliance G O V E R N A N C E : W H O A C C E S S E D W H A T Unity Catalog: Library Security TheDataLead AI -
Instant visibility into trends & performance Better decisions with real-time
data Simplifies complexity for all users Monitors KPIs like late returns or popular books Boosts engagement across teams DASHBOARD VISUALIZATION TheDataLead AI Your visual storytelling -
W H Y T H I S M A T
T E R S F O R Y O U R B U S I N E S S Lower costs -> one platform Faster answers Smarter decisions Happier teams -> one unified tool Why Smart Libraries Win TheDataLead AI -
W H A T Y O U L E A
R N E D What a Lakehouse is and why it matters How to structure pipelines using Medallion architecture Hands-on SQL with real-life data K E Y T A K E A W A Y S TheDataLead AI -
TheDataLead AI Who We Are? TheDataLead AI is a UK-based
consultancy focused exclusively on delivering high-impact solutions with Databricks. We help budget concious organisations unlock the full value of their data through advanced analytics, AI and ML, and scalable data infrastructure. -
TheDataLead AI We’d love to hear from you! A short
feedback form will be sent to your email shortly. We'd really appreciate you taking a moment to share your thoughts. -
Thank you! TheDataLead AI www.thedatalead.ai Follow us on LinkedIn for
more content on Databricks Lakehouse www.linkedin.com/company/the-data-lead/ -
Enjoyed the session? Share your key takeaway on LinkedIn and
tag us! We’d love to hear your thoughts! TheDataLead AI