Save 37% off PRO during our Black Friday Sale! »

Understanding the Basics of Data Analysis

Ee5407f7a79eb620c4fd54c136847b33?s=47 Piyush Verma
November 13, 2017
Technology
0
120

Understanding the Basics of Data Analysis

Read more on http://blog.oogway.in

Ee5407f7a79eb620c4fd54c136847b33?s=128

Piyush Verma

November 13, 2017
Tweet

More Decks by Piyush Verma

See All by Piyush Verma
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
54
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
86
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
490
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
1
410
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
46
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
60
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
25
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
38
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
47

Other Decks in Technology

See All in Technology
Bc76bcaa1b60adef57995ba63697bdee?s=48 hiroyukiyagihashi
0
440
A857277d74d4719f7f7751dca5ed553e?s=48 cmusudakeisuke
0
600
7b1f62709e6b808406596d8676f7f09c?s=48 udayan28
0
120
81902928da54ee11fd4028144d061993?s=48 tohae
1
210
88b514e6413838c0d0cfb3257cefd19d?s=48 kamata_shingo
3
630
B403e8d81063fed04df41aa00941aced?s=48 coopsapporo
1
190
69f5ab96892df4d8cbe32189f2ed2d9d?s=48 heyinc
2
7.4k
997716e957fe4db19491b2794346a7ce?s=48 airreader
0
280
506d54f33118160c5d41f73d223544d0?s=48 wataru420
0
170
0e3cc0e218e5bcb4c4da827fb5668f5a?s=48 stopendy
1
140
23f4d5d797a91b6d17d627b90b5a42d9?s=48 kentaro
0
440
C825832c4fc71ffdfd44905729281fb0?s=48 puhitaku
0
2.8k

Featured

See All Featured
D36d2c1821af9249b69ff7f5ed60529b?s=48 tammielis
241
23k
Ef32e0b1ac5082450dc8c1fca3a26b5c?s=48 cherdarchuk
78
280k
5f2da528927a2ec9ba4fec2069cbc958?s=48 kneath
298
39k
56c4053438af8e8b90d6f53cbb7573be?s=48 jakevdp
779
200k
32de0bd2ba869609d26fd052a4622778?s=48 cromwellryan
107
6.6k
Ae14cc4491ac334f9cd23f9f93b4305e?s=48 orderedlist
PRO
332
36k
465724d73fe3a92c0879fdfb43a3a6f3?s=48 philhawksworth
195
9.1k
7cbd3f5f922d798cc312eaf596de7ae6?s=48 iamctodd
33
3.1k
48c098b6579220a67a6ba949be6e4b5a?s=48 revolveconf
204
10k
D83926c323d4f9289f947b4b4e76b939?s=48 jensimmons
210
11k
78b475797a14c84799063c7cd073962f?s=48 holman
451
140k
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
25
960

Transcript

  1. ABC of Distributed Data Processing. Achieving Buzzword Compliance. 1 Piyush

    Verma Oogway Consulting

  2. Common thoughts 2

  3. When will my Data become Big Data?

  4. Hive Data Will Save.

  5. How did we reach here? 5

  6. Data :: Business

  7. Data :: Business

  8. Types of Workload

  9. When do I call it Big Enough?

  10. Why bother with Data Engineering? 10

  11. Why do analysis at all?

  12. Descriptive - Historical. - Deterministic. - Inferential. - Managers make

    pretty graphs.

  13. Predictive - Future. - Probabilistic. - Based on Descriptive. -

    This is what armchair critics do.

  14. Prescriptive

  15. Architecture: Round 1 15

  16. What does data look like?

  17. Storage Choice 1

  18. Storage Choice 2

  19. Challenges: Round 1 19

  20. Scaling

  21. Archival Policy

  22. Oh no

  23. Garbage / Purging

  24. All related entities end up in complex joins

  25. All Relationships complicate over Dimension of time

  26. Anatomy 26

  27. Anatomy

  28. Challenges: Round 2 28

  29. Snowflake Schema

  30. Star Schema

  31. De-Duplication

  32. Bloom Filters Cuckoo Filters - Does not exist for sure.

    - May or may not exist.

  33. Slow Changing Dimensions

  34. Batching vs Streaming

  35. Out-of-Order Processing

  36. Cubes • Efficiency of Retrieval • Warehouse:Cube :: DB:Table •

    View: Dimension + Measure • Slice, Dice & Rotate

  37. Architecture: Revisited 37

  38. Sample Solution

  39. Thank you! Piyush Verma @meson10 Oogway Consulting http://oogway.in