Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding the Basics of Data Analysis

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Piyush Verma Piyush Verma
November 13, 2017
Technology
350
0

Understanding the Basics of Data Analysis

Read more on http://blog.oogway.in

Avatar for Piyush Verma

Piyush Verma

November 13, 2017

More Decks by Piyush Verma

See All by Piyush Verma
SLOs that Lie
Avatar for Piyush Verma meson10
0
130
Doing SRE the right way - 2
Avatar for Piyush Verma meson10
0
170
Doing SRE the right way
Avatar for Piyush Verma meson10
0
1k
Observability and Control Theory
Avatar for Piyush Verma meson10
1
1.1k
Reliability
Avatar for Piyush Verma meson10
0
170
Reliability of Distributed Systems
Avatar for Piyush Verma meson10
0
290
My TLS was broken
Avatar for Piyush Verma meson10
0
150
Technology that builds Organizations
Avatar for Piyush Verma meson10
0
150
Namespace.go
Avatar for Piyush Verma meson10
0
180

Other Decks in Technology

See All in Technology
（きっとたぶん）人材育成や教育のような何かの話
Avatar for Takanori Sejima sejima
0
720
"うちにはまだ早い"は本当？ ─ 小さく始めるPlatform Engineering入門
Avatar for Haruka Sakihara harukasakihara
6
520
『生成AI時代のクレデンシャルとパーミッション設計 — Claude Code を起点に』の執筆企画
Avatar for Takuro SASAKI takuros
3
2.3k
ESP32 IoTを動かしながらメモリ使用量を観測してみた話
Avatar for ZOZO Developers zozotech
PRO
0
110
PdM・Eng・QAで進めるAI駆動開発の現在地/aidd-with-pdm-eng-qa
Avatar for butatamasoba / Shota Kusaba / 草場翔太 shota_kusaba
0
210
カオナビに Suspenseを導入するまで / The Road to Suspense at kaonavi
Avatar for 株式会社カオナビ kaonavi
1
450
AI-Assisted Contributions and Maintainer Load - PyCon US 2026
Avatar for Paolo Melchiorre pauloxnet
1
110
React 19×Rustツール 進化の「ズレ」を設計で埋める
Avatar for Hayashi Takao remrem0090
1
110
20260507-ACL-seminar
Avatar for Satoshi5884 satoshi5884
0
110
マンション備え付けのネットワークとLTE回線を組み合わせた ネットワークの安定化の考案
Avatar for harutiro harutiro
1
120
SREの仕事は「壊さないこと」ではなくなった 〜自律化していくシステムに、責任と判断を与えるという価値〜 / 20260515 Naoki Shimada
Avatar for SHIFT EVOLVE shift_evolve
PRO
1
140
会社説明資料｜株式会社ギークプラス ソフトウェア事業部
Avatar for ギークプラス ソフトウェア事業部 geekplus_tech
0
220

Featured

See All Featured
Prompt Engineering for Job Search
Avatar for Mfonobong Umondia mfonobong
0
300
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
66
8.4k
How to audit for AI Accessibility on your Front & Back End
Avatar for davetheseo davetheseo
0
360
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
275
41k
Self-Hosted WebAssembly Runtime for Runtime-Neutral Checkpoint/Restore in Edge–Cloud Continuum
Avatar for Yuki Nakata chikuwait chikuwait
0
510
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
13
1.2k
No one is an island. Learnings from fostering a developers community.
Avatar for Antonio thoeni
21
3.7k
Introduction to Domain-Driven Design and Collaborative software design
Avatar for Kenny Baas-Schwegler baasie
1
780
Ten Tips & Tricks for a 🌱 transition
Avatar for Manu Carrasco Molina stuffmc
0
110
HTML-Aware ERB: The Path to Reactive Rendering @ RubyCon 2026, Rimini, Italy
Avatar for Marco Roth marcoroth
1
36
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
270
14k
Collaborative Software Design: How to facilitate domain modelling decisions
Avatar for Kenny Baas-Schwegler baasie
1
210

Transcript

  1. ABC of Distributed Data Processing. Achieving Buzzword Compliance. 1 Piyush

    Verma Oogway Consulting

  2. Common thoughts 2

  3. When will my Data become Big Data?

  4. Hive Data Will Save.

  5. How did we reach here? 5

  6. Data :: Business

  7. Data :: Business

  8. Types of Workload

  9. When do I call it Big Enough?

  10. Why bother with Data Engineering? 10

  11. Why do analysis at all?

  12. Descriptive - Historical. - Deterministic. - Inferential. - Managers make

    pretty graphs.

  13. Predictive - Future. - Probabilistic. - Based on Descriptive. -

    This is what armchair critics do.

  14. Prescriptive

  15. Architecture: Round 1 15

  16. What does data look like?

  17. Storage Choice 1

  18. Storage Choice 2

  19. Challenges: Round 1 19

  20. Scaling

  21. Archival Policy

  22. Oh no

  23. Garbage / Purging

  24. All related entities end up in complex joins

  25. All Relationships complicate over Dimension of time

  26. Anatomy 26

  27. Anatomy

  28. Challenges: Round 2 28

  29. Snowflake Schema

  30. Star Schema

  31. De-Duplication

  32. Bloom Filters Cuckoo Filters - Does not exist for sure.

    - May or may not exist.

  33. Slow Changing Dimensions

  34. Batching vs Streaming

  35. Out-of-Order Processing

  36. Cubes • Efficiency of Retrieval • Warehouse:Cube :: DB:Table •

    View: Dimension + Measure • Slice, Dice & Rotate

  37. Architecture: Revisited 37

  38. Sample Solution

  39. Thank you! Piyush Verma @meson10 Oogway Consulting http://oogway.in