Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding the Basics of Data Analysis

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Piyush Verma Piyush Verma
November 13, 2017
Technology
350
0

Understanding the Basics of Data Analysis

Read more on http://blog.oogway.in

Avatar for Piyush Verma

Piyush Verma

November 13, 2017

More Decks by Piyush Verma

See All by Piyush Verma
SLOs that Lie
Avatar for Piyush Verma meson10
0
120
Doing SRE the right way - 2
Avatar for Piyush Verma meson10
0
170
Doing SRE the right way
Avatar for Piyush Verma meson10
0
1k
Observability and Control Theory
Avatar for Piyush Verma meson10
1
1.1k
Reliability
Avatar for Piyush Verma meson10
0
160
Reliability of Distributed Systems
Avatar for Piyush Verma meson10
0
280
My TLS was broken
Avatar for Piyush Verma meson10
0
150
Technology that builds Organizations
Avatar for Piyush Verma meson10
0
150
Namespace.go
Avatar for Piyush Verma meson10
0
180

Other Decks in Technology

See All in Technology
コミュニティ・勉強会を作るのは目的じゃない
Avatar for uutan1108 ohmori_yusuke
0
280
需要創出(Chatwork)×供給(BPaaS) フライホイールとMoat 実行能力の最適配置とAI戦略
Avatar for kubell kubell_hr
0
1.4k
AWS Agent Registry の基礎・概要を理解する/aws-agent-registry-intro
Avatar for Renya Kujirada ren8k
3
420
AI時代 に増える データ活用先
Avatar for ono.takayuki takahal
0
340
Oracle AI Database@AWS：サービス概要のご紹介
Avatar for oracle4engineer oracle4engineer
PRO
4
2.4k
Percolatorを廃止し、マルチ検索サービスへ刷新した話 / Search Engineering Tech Talk 2026 Spring
Avatar for Visional Engineering ＆ Design visional_engineering_and_design
0
200
AIが書いたコードを信じられない問題 〜レビュー負荷を下げるために変えたこと〜 / The AI Code Trust Gap: Reducing the Review Burden
Avatar for 株式会社ビットキー / Bitkey Inc. bitkey
PRO
8
1.4k
EMから幅を広げるために最近挑戦していること / Recent challenges I'm undertaking to expand my horizons beyond EM
Avatar for hiro-torii hiro_torii
1
160
20年前の「OSS革命」に学ぶ AI時代の生存戦略
Avatar for Sam Akada samakada
0
500
要件定義の精度を高めるための型と生成AIの活用 / Using Types and Generative AI to Improve the Accuracy of Requirements Definition
Avatar for haru860 haru860
0
110
Anthropic「Long-running a gents」をGeminiで再現してみた
Avatar for t-kikuchi tkikuchi
0
730
MySQL 9.7がやってきた ～これまでのあらすじと基本情報～ @ 日本MySQLユーザ会会2026年04月 / mysql97-yattekita
Avatar for sakaik sakaik
0
120

Featured

See All Featured
Practical Tips for Bootstrapping Information Extraction Pipelines
Avatar for Matthew Honnibal honnibal
25
1.9k
Abbi's Birthday
Avatar for Violet Bock coloredviolet
2
7.3k
Beyond borders and beyond the search box: How to win the global "messy middle" with AI-driven SEO
Avatar for David Carrasco davidcarrasco
3
120
10 Git Anti Patterns You Should be Aware of
Avatar for Lemi Orhan Ergin lemiorhan
PRO
659
62k
How People are Using Generative and Agentic AI to Supercharge Their Products, Projects, Services and Value Streams Today
Avatar for Helen Beal helenjbeal
1
170
Leveraging LLMs for student feedback in introductory data science courses - posit::conf(2025)
Avatar for Mine Cetinkaya-Rundel minecr
1
240
Discover your Explorer Soul
Avatar for Emna emna__ayadi
2
1.1k
WCS-LA-2024
Avatar for Leonardo Collado-Torres lcolladotor
0
550
[SF Ruby Conf 2025] Rails X
Avatar for Vladimir Dementyev palkan
2
980
Organizational Design Perspectives: An Ontology of Organizational Design Elements
Avatar for Dr. Kim W Petersen kimpetersen
PRO
1
680
The SEO identity crisis: Don't let AI make you average
Avatar for Varn varn
0
450
Collaborative Software Design: How to facilitate domain modelling decisions
Avatar for Kenny Baas-Schwegler baasie
1
200

Transcript

  1. ABC of Distributed Data Processing. Achieving Buzzword Compliance. 1 Piyush

    Verma Oogway Consulting

  2. Common thoughts 2

  3. When will my Data become Big Data?

  4. Hive Data Will Save.

  5. How did we reach here? 5

  6. Data :: Business

  7. Data :: Business

  8. Types of Workload

  9. When do I call it Big Enough?

  10. Why bother with Data Engineering? 10

  11. Why do analysis at all?

  12. Descriptive - Historical. - Deterministic. - Inferential. - Managers make

    pretty graphs.

  13. Predictive - Future. - Probabilistic. - Based on Descriptive. -

    This is what armchair critics do.

  14. Prescriptive

  15. Architecture: Round 1 15

  16. What does data look like?

  17. Storage Choice 1

  18. Storage Choice 2

  19. Challenges: Round 1 19

  20. Scaling

  21. Archival Policy

  22. Oh no

  23. Garbage / Purging

  24. All related entities end up in complex joins

  25. All Relationships complicate over Dimension of time

  26. Anatomy 26

  27. Anatomy

  28. Challenges: Round 2 28

  29. Snowflake Schema

  30. Star Schema

  31. De-Duplication

  32. Bloom Filters Cuckoo Filters - Does not exist for sure.

    - May or may not exist.

  33. Slow Changing Dimensions

  34. Batching vs Streaming

  35. Out-of-Order Processing

  36. Cubes • Efficiency of Retrieval • Warehouse:Cube :: DB:Table •

    View: Dimension + Measure • Slice, Dice & Rotate

  37. Architecture: Revisited 37

  38. Sample Solution

  39. Thank you! Piyush Verma @meson10 Oogway Consulting http://oogway.in