Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding the Basics of Data Analysis

Ee5407f7a79eb620c4fd54c136847b33?s=47 Piyush Verma
November 13, 2017
Technology
0
110

Understanding the Basics of Data Analysis

Read more on http://blog.oogway.in

Ee5407f7a79eb620c4fd54c136847b33?s=128

Piyush Verma

November 13, 2017
Tweet

More Decks by Piyush Verma

See All by Piyush Verma
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
52
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
82
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
370
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
340
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
41
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
58
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
22
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
36
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
47

Other Decks in Technology

See All in Technology
3f848c0c0b9a6603894e157da93e4655?s=48 sadayoshitada0919
0
1.4k
3e8f61263f14a620f21d5f3f89c4b846?s=48 gamella
1
1.3k
F04edc7cb2099745e5413c754d3d22f5?s=48 benoitc
0
100
4924e08c88a442edbffd87b91cb45131?s=48 zaki_lknr
0
510
2219597a939c895e5db6f8b7c94e2859?s=48 pojiro
1
120
842515eaf8fbb2dfcc75197e7797dc15?s=48 sat
0
360
Fd38d7ee40295f6361aa059008288731?s=48 shyaginuma
14
5.1k
4e7f712e4bba241f579d3dc0eaf130d4?s=48 mito201
0
370
97b3cca999b52cb5296675ac0a5c12cd?s=48 emiki
2
220
6ddc65998177a0f05be629dc31321a8f?s=48 kken78
0
150
C4c161ae9eeeed8f161197410f7a228a?s=48 sizuhiko
0
390
4ec53f1dd4a246281d4cc0bec20c80bd?s=48 yomaki
0
340

Featured

See All Featured
D83926c323d4f9289f947b4b4e76b939?s=48 jensimmons
209
10k
9375a9529679f1b42b567a640d775e7d?s=48 schacon
152
6.8k
Ba530e786553390f3fb271848485681c?s=48 3n
167
22k
3d4519fd34b76fb265fc0237f3792bd4?s=48 danielanewman
208
20k
2f5463832ccb768ccb4a1ca3607c27ef?s=48 jacobian
260
20k
4394ceb8b277f35ee3da7030384efb5d?s=48 davidbonilla
75
3.7k
170b760e0147792d0140e59ec78e9893?s=48 eitanlees
118
10k
56c4053438af8e8b90d6f53cbb7573be?s=48 jakevdp
777
200k
245cee81a9c424266e5e401d844ea881?s=48 lara
26
3.2k
6804f1775cb4babfcc3851298566fbce?s=48 tanoku
91
8.8k
5f50f94346fbcb23706f0707169317a4?s=48 aarron
261
36k
88bc30284c6a424b63a92aad27d0ba36?s=48 jnunemaker
PRO
43
4.8k

Transcript

  1. ABC of Distributed Data Processing. Achieving Buzzword Compliance. 1 Piyush

    Verma Oogway Consulting

  2. Common thoughts 2

  3. When will my Data become Big Data?

  4. Hive Data Will Save.

  5. How did we reach here? 5

  6. Data :: Business

  7. Data :: Business

  8. Types of Workload

  9. When do I call it Big Enough?

  10. Why bother with Data Engineering? 10

  11. Why do analysis at all?

  12. Descriptive - Historical. - Deterministic. - Inferential. - Managers make

    pretty graphs.

  13. Predictive - Future. - Probabilistic. - Based on Descriptive. -

    This is what armchair critics do.

  14. Prescriptive

  15. Architecture: Round 1 15

  16. What does data look like?

  17. Storage Choice 1

  18. Storage Choice 2

  19. Challenges: Round 1 19

  20. Scaling

  21. Archival Policy

  22. Oh no

  23. Garbage / Purging

  24. All related entities end up in complex joins

  25. All Relationships complicate over Dimension of time

  26. Anatomy 26

  27. Anatomy

  28. Challenges: Round 2 28

  29. Snowflake Schema

  30. Star Schema

  31. De-Duplication

  32. Bloom Filters Cuckoo Filters - Does not exist for sure.

    - May or may not exist.

  33. Slow Changing Dimensions

  34. Batching vs Streaming

  35. Out-of-Order Processing

  36. Cubes • Efficiency of Retrieval • Warehouse:Cube :: DB:Table •

    View: Dimension + Measure • Slice, Dice & Rotate

  37. Architecture: Revisited 37

  38. Sample Solution

  39. Thank you! Piyush Verma @meson10 Oogway Consulting http://oogway.in