Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From Zero to Lakehouse

From Zero to Lakehouse

Avatar for Paul Karikari

Paul Karikari

August 07, 2025

More Decks by Paul Karikari

Other Decks in Technology

Transcript

  1. F R O M Z E R O T O

    L A K E H O U S E FIRST STEPS WITH DATABRICKS Paul Karikari Principal Solutions Architect- Data & AI, TheDataLead AI 27 June, 2025 TheDataLead AI
  2. W E L C O M E & A G

    E N D A Introduction Data Management The Library Analogy Lakehouse Platform Overview Hands-On Demo Q&A + Wrap-Up T A B L E O F C O N T E N T S TheDataLead AI
  3. "Without data, you're just another person with a Wi-Fi connection

    and a strong opinion." 😄 TheDataLead AI
  4. Why do businesses NEED data? TheDataLead AI Because "gut instinct"

    doesn’t scale past five people and a Slack channel. Because guessing your customer’s needs is so 1995. Helping businesses figure out what’s actually happening… Instead of just arguing about it in long meetings.
  5. W H Y D O B U S I N

    E S S E S N E E D D A T A ? TheDataLead AI 1. Understanding Customer Behaviour Track preferences, behaviour, and feedback. Example: An e-commerce company uses purchase history to recommend products.
  6. TheDataLead AI 2. Improving Products and Services Identify what works

    and what doesn’t. Use data from usage patterns, error reports, and surveys to improve features. W H Y D O B U S I N E S S E S N E E D D A T A ?
  7. TheDataLead AI 3. Better Decision-Making Replace gut-feeling with evidence-based strategies.

    Data helps forecast trends, plan budgets, and choose where to invest. W H Y D O B U S I N E S S E S N E E D D A T A ?
  8. TheDataLead AI 4. Optimising Operations Improve efficiency in supply chains,

    staffing, or manufacturing. Example: Analysing delivery routes to reduce shipping time and cost. W H Y D O B U S I N E S S E S N E E D D A T A ?
  9. TheDataLead AI 5. Staying Competitive Benchmark against industry data. Spot

    opportunities and threats earlier than rivals. W H Y D O B U S I N E S S E S N E E D D A T A ?
  10. TheDataLead AI 6. Compliance and Reporting Businesses need data to

    meet legal, tax, and regulatory requirements. Example: Financial records, GDPR audit trails. W H Y D O B U S I N E S S E S N E E D D A T A ?
  11. TheDataLead AI 7. Driving Innovation Power machine learning, personalisation, and

    automation. Data fuels new business models and smarter experiences (e.g., Netflix recommendations, Tesla's self-driving algorithms). W H Y D O B U S I N E S S E S N E E D D A T A ?
  12. T H E D A T A C H A

    L L E N G E TheDataLead AI Businesses are flooded with data but struggle to turn it into decisions. One team wants raw data fast. Another needs trusted, curated insights. Juggling costs, access, and governance.
  13. E V O L U T I O N O

    F D A T A S T O R A G E S Y S T E M S TheDataLead AI D I S K S T O R A G E R E L A T I O N A L D B M S D A TA WA R E H O U S E D A TA L A K E L A K E H O U S E
  14. OLTP: Like a cash register – fast, accurate, handles lots

    of small transactions. OLAP: Like a reporting dashboard – tells you items sold, when, and why. TheDataLead AI
  15. Why did OLTP break up with OLAP? Because every time

    OLTP wanted to keep things quick and transactional, OLAP wanted to “analyze the relationship” from every possible angle. 😅 TheDataLead AI
  16. Books Catalog Librarians Readers Shelves Governance T H E W

    O R L D O F D A T A I S L I K E A L I B R A R Y What If Data Were Like a Library? Just like a library has structure, organisation, and purpose, so should your data. TheDataLead AI Raw data files (CSV, JSON) Metadata tables and schemas Data engineers maintaining order Business users & analysts Storage layers Library Policies
  17. DATA WAREHOUSE : THE CURATED ROOM T R A D

    I T I O N A L D A T A W A R E H O U S E S TheDataLead AI T H E L I B R A R Y A N A L O G Y Once upon a time, knowledge was kept in a physical library. Only clean, pre-approved books could enter. Everything was in order until someone asked for a rough draft or a sketch. The system worked well, but it was inflexible.
  18. DATA WAREHOUSE : THE CURATED ROOM Curated, cleaned content only

    Limited types of books allowed High performance, but rigid T R A D I T I O N A L D A T A W A R E H O U S E S TheDataLead AI T H E L I B R A R Y A N A L O G Y
  19. TheDataLead AI The polished library had rules. But soon, businesses

    needed messy notes, images, recordings data that didn’t fit into tidy rows. They needed instant updates, machine learning experiments, real-time insights. The library couldn’t keep up. T H E L I B R A R Y A N A L O G Y L I M I T A T I O N S O F T R A D I T I O N A L D A T A W A R E H O U S E S WHY THE LIBRARY MODEL BROKE DOWN
  20. TheDataLead AI Limited types of books allowed - Structured Data

    only Slow to update means delayed data availability Not Built for AI/ML because no access to raw, historical or unstructured data. T H E L I B R A R Y A N A L O G Y WHY THE LIBRARY MODEL BROKE DOWN L I M I T A T I O N S O F T R A D I T I O N A L D A T A W A R E H O U S E S
  21. DATA LAKE : THE BASEMENT ARCHIVE T R A D

    I T I O N A L D A T A L A K E S TheDataLead AI In frustration, the city built a giant basement to hold everything. No rules, no structure just piles of books, napkins, and notes. It was cheap to build, but impossible to navigate. Insights were buried under chaos
  22. DATA LAKE : THE BASEMENT ARCHIVE All data dumped in

    raw form Great for storage, not for access No structure, hard to find anything T R A D I T I O N A L D A T A L A K E S TheDataLead AI
  23. E N T E R T H E L A

    K E H O U S E : A S M A R T , U N I F I E D L I B R A R Y T H E B E S T O F B O T H W O R L D S TheDataLead AI Then came a new kind of library which was open yet organized. It welcomed raw ideas and polished books. Auto-sorting shelves and robotic assistants. Fast, smart, and flexible. This called the Lakehouse
  24. W H A T I S T H E D

    A T A B R I C K S L A K E H O U S E ? O N E U N I F I E D P L A T F O R M Combines the structure of warehouses with the scalability of data lakes Built on Delta Lake (open source) Supports SQL, Python, ML, streaming, and more Perfect for collaboration and governance TheDataLead AI
  25. M E D A L L I O N A

    R C H I T E C T U R E A logical organization of data in a Lakehouse TheDataLead AI
  26. Component Analogy Role Delta Lake Smart bookshelves ACID storage, versioning

    Apache Spark Fast librarians Scans 1000s of books instantly MLflow Data scientist’s notebook Tracks predictions and experiments Unity Catalog Access control panel Who read what, and when K E Y C O M P O N E N T S Your Smart Library Staff TheDataLead AI
  27. D A T A M O D E L :

    L I B R A R Y S Y S T E M TheDataLead AI Structure & Clarity : Organizes data into understandable formats. Business Alignment : Enforces consistent definitions and rules. Reusability : Enables multiple teams to build on clean, reliable data. Scalability : Makes it easy to grow the pipeline or add new features. Governance : Supports compliance, documentation, and data lineage Why a Data Model Matters
  28. Keeps logs of all readers and access Controls who can

    borrow special collections (e.g., sensitive data) Helps with audits and compliance G O V E R N A N C E : W H O A C C E S S E D W H A T Unity Catalog: Library Security TheDataLead AI
  29. Instant visibility into trends & performance Better decisions with real-time

    data Simplifies complexity for all users Monitors KPIs like late returns or popular books Boosts engagement across teams DASHBOARD VISUALIZATION TheDataLead AI Your visual storytelling
  30. W H Y T H I S M A T

    T E R S F O R Y O U R B U S I N E S S Lower costs -> one platform Faster answers Smarter decisions Happier teams -> one unified tool Why Smart Libraries Win TheDataLead AI
  31. W H A T Y O U L E A

    R N E D What a Lakehouse is and why it matters How to structure pipelines using Medallion architecture Hands-on SQL with real-life data K E Y T A K E A W A Y S TheDataLead AI
  32. TheDataLead AI Who We Are? TheDataLead AI is a UK-based

    consultancy focused exclusively on delivering high-impact solutions with Databricks. We help budget concious organisations unlock the full value of their data through advanced analytics, AI and ML, and scalable data infrastructure.
  33. TheDataLead AI We’d love to hear from you! A short

    feedback form will be sent to your email shortly. We'd really appreciate you taking a moment to share your thoughts.
  34. Thank you! TheDataLead AI www.thedatalead.ai Follow us on LinkedIn for

    more content on Databricks Lakehouse www.linkedin.com/company/the-data-lead/
  35. Enjoyed the session? Share your key takeaway on LinkedIn and

    tag us! We’d love to hear your thoughts! TheDataLead AI