Understanding the Basics of Data Analysis

Ee5407f7a79eb620c4fd54c136847b33?s=47 Piyush Verma
November 13, 2017
Technology
0
94

Understanding the Basics of Data Analysis

Read more on http://blog.oogway.in

Ee5407f7a79eb620c4fd54c136847b33?s=128

Piyush Verma

November 13, 2017
Tweet

More Decks by Piyush Verma

See All by Piyush Verma
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
30
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
50
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
26
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
180
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
29
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
41
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
14
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
17
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
38

Other Decks in Technology

See All in Technology
C786d7772602b488a7972c38c143f61c?s=48 con_mame
3
1.8k
F01c8fdfa64bdfc00a94896e0b1359e0?s=48 tmdoi
0
240
Ac85568dfa460d48c36095d142632b1a?s=48 wps
0
170
Cd931bc05e7cee46b9a950a63e47ba4c?s=48 you
0
120
8120d39cb575881c830287046f06fec8?s=48 daoka
1
2k
2102a6b8760bd6f57f672805723dd83a?s=48 line_developers_tw
0
840
36bb9dcd778d2a9621d44e92425d0907?s=48 hhiroshell
1
340
74cec195bfb6cb5165256d88cb7fcf0f?s=48 miu_crescent
5
890
404b6b310d5fc7680a62ef09a463a794?s=48 yutakakinjyo
0
350
11b879ad001638b95dd62331eed65505?s=48 fortegp05
0
230
332f89cc697355902a817506b6995f2b?s=48 ytaka23
2
710
Cb7edd153cf503f44c727b73647558bf?s=48 tetsuroito
0
910

Featured

See All Featured
F68cb25ae3e5c2106997454b13b7737d?s=48 brad_frost
154
6.2k
7c8469ee8c9e594c65c59b919626c08d?s=48 scottboms
250
11k
A0517b08532aec8ab2aae3f65d40ad86?s=48 hannesfritz
26
870
B47b288784fda77a94aeb234ca743c24?s=48 keavy
104
14k
4262b9cfacfb6b2c77f23e1d0310cd3e?s=48 myddelton
104
10k
B3ace07412a5178ea778e7eec161fc0a?s=48 jcasabona
7
410
245cee81a9c424266e5e401d844ea881?s=48 lara
168
9.2k
766b9746b59e6f09e280cd33cf4ed419?s=48 skipperchong
4
490
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
22
1.6k
96270e4c3e5e9806cf7245475c00b275?s=48 addyosmani
308
21k
F383c6a4dc55e331bbe2987b622cee6b?s=48 stephaniewalter
256
10k
9fa239b3154a0169d99ab7bf1422dd12?s=48 marcelosomers
220
14k

Transcript

  1. ABC of Distributed Data Processing. Achieving Buzzword Compliance. 1 Piyush

    Verma Oogway Consulting

  2. Common thoughts 2

  3. When will my Data become Big Data?

  4. Hive Data Will Save.

  5. How did we reach here? 5

  6. Data :: Business

  7. Data :: Business

  8. Types of Workload

  9. When do I call it Big Enough?

  10. Why bother with Data Engineering? 10

  11. Why do analysis at all?

  12. Descriptive - Historical. - Deterministic. - Inferential. - Managers make

    pretty graphs.

  13. Predictive - Future. - Probabilistic. - Based on Descriptive. -

    This is what armchair critics do.

  14. Prescriptive

  15. Architecture: Round 1 15

  16. What does data look like?

  17. Storage Choice 1

  18. Storage Choice 2

  19. Challenges: Round 1 19

  20. Scaling

  21. Archival Policy

  22. Oh no

  23. Garbage / Purging

  24. All related entities end up in complex joins

  25. All Relationships complicate over Dimension of time

  26. Anatomy 26

  27. Anatomy

  28. Challenges: Round 2 28

  29. Snowflake Schema

  30. Star Schema

  31. De-Duplication

  32. Bloom Filters Cuckoo Filters - Does not exist for sure.

    - May or may not exist.

  33. Slow Changing Dimensions

  34. Batching vs Streaming

  35. Out-of-Order Processing

  36. Cubes • Efficiency of Retrieval • Warehouse:Cube :: DB:Table •

    View: Dimension + Measure • Slice, Dice & Rotate

  37. Architecture: Revisited 37

  38. Sample Solution

  39. Thank you! Piyush Verma @meson10 Oogway Consulting http://oogway.in