Understanding the Basics of Data Analysis

Ee5407f7a79eb620c4fd54c136847b33?s=47 Piyush Verma
November 13, 2017
Technology
0
100

Understanding the Basics of Data Analysis

Read more on http://blog.oogway.in

Ee5407f7a79eb620c4fd54c136847b33?s=128

Piyush Verma

November 13, 2017
Tweet

More Decks by Piyush Verma

See All by Piyush Verma
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
32
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
58
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
73
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
220
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
32
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
46
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
16
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
22
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
39

Other Decks in Technology

See All in Technology
73560128b23de542e47a318145bc781a?s=48 kawanamiyuu
0
230
404139d782ec666acea93dffc86e089f?s=48 sylph01
0
170
7cd783377515bdf8207062840b7b2f4e?s=48 soracom
1
130
0eacc4d25ee36af36d12e76b52de8bac?s=48 koushiohno
0
130
22ac00b36d2057e61ee1c06f0f5dc6cb?s=48 nakazax
0
460
73c174b34dafaea64f2824eb008a6559?s=48 shibayan
0
1.1k
0d040d77cca56a9652d74db65bac0245?s=48 elasticanz
0
130
7cd783377515bdf8207062840b7b2f4e?s=48 soracom
0
250
7e20183342a040a32924a41343603286?s=48 konabuta
0
490
96e6721cf9462dfc15a564189a701d29?s=48 naoyasugita
0
130
7cd783377515bdf8207062840b7b2f4e?s=48 soracom
0
1.1k
2dfb65b481d9e382551b76c7cb0b7c98?s=48 ishiagme
1
190

Featured

See All Featured
3d4519fd34b76fb265fc0237f3792bd4?s=48 danielanewman
1
400
7fa6101cd43067653cf4ac6c7ca54305?s=48 akmur
252
19k
C9b18d9dff88a9dd2393364c2b3b21bd?s=48 colly
66
2.9k
2f5463832ccb768ccb4a1ca3607c27ef?s=48 jacobian
255
19k
11c84dff30266b907714802088852677?s=48 jasonvnalue
82
8k
7cbd3f5f922d798cc312eaf596de7ae6?s=48 iamctodd
13
1.7k
25c7c18223fb42a4c6ae1c8db6f50f9b?s=48 mojombo
358
62k
Eb8975af8e49e19e3dd6b6b84a542e26?s=48 qrush
283
18k
5f83ef806155b403c3393160ce51f955?s=48 jlugia
216
16k
C620790ae5bf5b50c245b2e0ef95f338?s=48 deanohume
295
27k
64827b31aef81498662e8af4bd6d5157?s=48 jonrohan
1020
370k
433acaea1012b25d97ae66da9b998dad?s=48 malarkey
393
60k

Transcript

  1. ABC of Distributed Data Processing. Achieving Buzzword Compliance. 1 Piyush

    Verma Oogway Consulting

  2. Common thoughts 2

  3. When will my Data become Big Data?

  4. Hive Data Will Save.

  5. How did we reach here? 5

  6. Data :: Business

  7. Data :: Business

  8. Types of Workload

  9. When do I call it Big Enough?

  10. Why bother with Data Engineering? 10

  11. Why do analysis at all?

  12. Descriptive - Historical. - Deterministic. - Inferential. - Managers make

    pretty graphs.

  13. Predictive - Future. - Probabilistic. - Based on Descriptive. -

    This is what armchair critics do.

  14. Prescriptive

  15. Architecture: Round 1 15

  16. What does data look like?

  17. Storage Choice 1

  18. Storage Choice 2

  19. Challenges: Round 1 19

  20. Scaling

  21. Archival Policy

  22. Oh no

  23. Garbage / Purging

  24. All related entities end up in complex joins

  25. All Relationships complicate over Dimension of time

  26. Anatomy 26

  27. Anatomy

  28. Challenges: Round 2 28

  29. Snowflake Schema

  30. Star Schema

  31. De-Duplication

  32. Bloom Filters Cuckoo Filters - Does not exist for sure.

    - May or may not exist.

  33. Slow Changing Dimensions

  34. Batching vs Streaming

  35. Out-of-Order Processing

  36. Cubes • Efficiency of Retrieval • Warehouse:Cube :: DB:Table •

    View: Dimension + Measure • Slice, Dice & Rotate

  37. Architecture: Revisited 37

  38. Sample Solution

  39. Thank you! Piyush Verma @meson10 Oogway Consulting http://oogway.in