Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding the Basics of Data Analysis

Ee5407f7a79eb620c4fd54c136847b33?s=47 Piyush Verma
November 13, 2017
Technology
0
110

Understanding the Basics of Data Analysis

Read more on http://blog.oogway.in

Ee5407f7a79eb620c4fd54c136847b33?s=128

Piyush Verma

November 13, 2017
Tweet

More Decks by Piyush Verma

See All by Piyush Verma
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
51
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
75
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
290
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
300
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
39
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
54
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
22
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
26
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
46

Other Decks in Technology

See All in Technology
9ea422a61c1a69c888786830eee3dbe6?s=48 osonoi
0
150
966e29b84ae7dbde170f66b0bc75b7a6?s=48 ishiayaya
PRO
0
170
2b680c5f22146e764e2998dd4595ec93?s=48 sue445
0
160
8fb8810e5e06a997bc13831b9b51007f?s=48 keinuma
0
250
6bb0559d2066b868dab09e435f488c6c?s=48 tomoyuki
0
530
54bcc1069d75636cc735e72877dd30ec?s=48 taichinakamura
0
1k
A97eee01397705443a72a48ce29d3e19?s=48 cybozuinsideout
4
3.2k
98f8e6125c052c4ac3fcc94f97919371?s=48 evilsmile
0
300
A84b3c763c9c543069b7c02551e2720e?s=48 yuyamada
0
510
Fc099d9851ad91908e0c16a70aa2ddc1?s=48 tikson
0
240
7cd783377515bdf8207062840b7b2f4e?s=48 soracom
0
470
05f8401462c2d4694f006634e970d577?s=48 ttakaguchi
0
130

Featured

See All Featured
F383c6a4dc55e331bbe2987b622cee6b?s=48 stephaniewalter
262
11k
Ef32e0b1ac5082450dc8c1fca3a26b5c?s=48 cherdarchuk
73
270k
9cde37f47e4a800ea081ea42de8d749a?s=48 dougneiner
120
8k
F716e5f737fcfb03f2cf8d1a184f1dbe?s=48 nonsquared
81
3.5k
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
28
2.2k
Fde6c3a50ca518a909ef35c22c0ee0e4?s=48 destraynor
222
47k
20bfe76b3d6105641f879fe45cfc9272?s=48 bkeepers
PRO
322
54k
5f50f94346fbcb23706f0707169317a4?s=48 aarron
258
36k
E6c6e133e74c3b83f04d2861deaa1c20?s=48 searls
204
38k
C9b18d9dff88a9dd2393364c2b3b21bd?s=48 colly
188
14k
B47b288784fda77a94aeb234ca743c24?s=48 keavy
108
14k
20bfe76b3d6105641f879fe45cfc9272?s=48 bkeepers
PRO
54
4.4k

Transcript

  1. ABC of Distributed Data Processing. Achieving Buzzword Compliance. 1 Piyush

    Verma Oogway Consulting

  2. Common thoughts 2

  3. When will my Data become Big Data?

  4. Hive Data Will Save.

  5. How did we reach here? 5

  6. Data :: Business

  7. Data :: Business

  8. Types of Workload

  9. When do I call it Big Enough?

  10. Why bother with Data Engineering? 10

  11. Why do analysis at all?

  12. Descriptive - Historical. - Deterministic. - Inferential. - Managers make

    pretty graphs.

  13. Predictive - Future. - Probabilistic. - Based on Descriptive. -

    This is what armchair critics do.

  14. Prescriptive

  15. Architecture: Round 1 15

  16. What does data look like?

  17. Storage Choice 1

  18. Storage Choice 2

  19. Challenges: Round 1 19

  20. Scaling

  21. Archival Policy

  22. Oh no

  23. Garbage / Purging

  24. All related entities end up in complex joins

  25. All Relationships complicate over Dimension of time

  26. Anatomy 26

  27. Anatomy

  28. Challenges: Round 2 28

  29. Snowflake Schema

  30. Star Schema

  31. De-Duplication

  32. Bloom Filters Cuckoo Filters - Does not exist for sure.

    - May or may not exist.

  33. Slow Changing Dimensions

  34. Batching vs Streaming

  35. Out-of-Order Processing

  36. Cubes • Efficiency of Retrieval • Warehouse:Cube :: DB:Table •

    View: Dimension + Measure • Slice, Dice & Rotate

  37. Architecture: Revisited 37

  38. Sample Solution

  39. Thank you! Piyush Verma @meson10 Oogway Consulting http://oogway.in