Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding the Basics of Data Analysis

Ee5407f7a79eb620c4fd54c136847b33?s=47 Piyush Verma
November 13, 2017
Technology
0
100

Understanding the Basics of Data Analysis

Read more on http://blog.oogway.in

Ee5407f7a79eb620c4fd54c136847b33?s=128

Piyush Verma

November 13, 2017
Tweet

More Decks by Piyush Verma

See All by Piyush Verma
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
41
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
67
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
190
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
280
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
37
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
48
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
17
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
22
Ee5407f7a79eb620c4fd54c136847b33?s=48 meson10
0
43

Other Decks in Technology

See All in Technology
9a883e4b99553b96d3ecf5a47aa6f233?s=48 khrd
1
660
A2cac4b3dcb2bc0b87917ddc034ef708?s=48 sansandsoc
0
470
877f92a3597118cda2715790fa170ad4?s=48 benzookapi
1
440
8a84268593355816432ceaf78777d585?s=48 dena_tech
0
170
Ad2155c75c67c0100dd008ad10858de5?s=48 legalforce
PRO
4
320
40301c0affdf359eaca771713e22b71a?s=48 harshbothra
0
150
B46a00cafe34a9437d3a5bc6afc5bee3?s=48 aditya45
2
2.4k
9e7a653ed313b3da4b263a772bfc5026?s=48 clustervr
0
250
4ccf6c02807d06f043a71435c48ce86a?s=48 eller86
1
240
D61ecb299008e313b7a375e186a0d6ae?s=48 gobeyond20xx
0
350
8611672fd60a86cd19c972c61df3c99f?s=48 900groove
2
530
1e8c5eaf563f965d3ba48b0054b1d955?s=48 shoichiron
1
150

Featured

See All Featured
282d599b414022eba5096510f675b6e2?s=48 chrislema
231
16k
E43f8dfd01057f8304f5f8fb9a294d7d?s=48 jeffersonlam
329
15k
Eb8975af8e49e19e3dd6b6b84a542e26?s=48 qrush
285
19k
053e75a5b48b44d6dd0612795dfb326d?s=48 notwaldorf
16
1.8k
A9179349dd2bdc67f377719f56d85656?s=48 garrettdimon
288
110k
7559f6cff1f5efc2d210965febd4d71c?s=48 bermonpainter
342
26k
E76911cbe088e5b850d966de3fc7435b?s=48 robhawkes
52
2.8k
78b475797a14c84799063c7cd073962f?s=48 holman
461
280k
61857dafbd287b3027c4dcea9008ad3c?s=48 carmenhchung
31
1.5k
24fc194843a71f10949be18d5a692682?s=48 gr2m
83
11k
Ae14cc4491ac334f9cd23f9f93b4305e?s=48 orderedlist
PRO
328
36k
E6c6e133e74c3b83f04d2861deaa1c20?s=48 searls
204
36k

Transcript

  1. ABC of Distributed Data Processing. Achieving Buzzword Compliance. 1 Piyush

    Verma Oogway Consulting

  2. Common thoughts 2

  3. When will my Data become Big Data?

  4. Hive Data Will Save.

  5. How did we reach here? 5

  6. Data :: Business

  7. Data :: Business

  8. Types of Workload

  9. When do I call it Big Enough?

  10. Why bother with Data Engineering? 10

  11. Why do analysis at all?

  12. Descriptive - Historical. - Deterministic. - Inferential. - Managers make

    pretty graphs.

  13. Predictive - Future. - Probabilistic. - Based on Descriptive. -

    This is what armchair critics do.

  14. Prescriptive

  15. Architecture: Round 1 15

  16. What does data look like?

  17. Storage Choice 1

  18. Storage Choice 2

  19. Challenges: Round 1 19

  20. Scaling

  21. Archival Policy

  22. Oh no

  23. Garbage / Purging

  24. All related entities end up in complex joins

  25. All Relationships complicate over Dimension of time

  26. Anatomy 26

  27. Anatomy

  28. Challenges: Round 2 28

  29. Snowflake Schema

  30. Star Schema

  31. De-Duplication

  32. Bloom Filters Cuckoo Filters - Does not exist for sure.

    - May or may not exist.

  33. Slow Changing Dimensions

  34. Batching vs Streaming

  35. Out-of-Order Processing

  36. Cubes • Efficiency of Retrieval • Warehouse:Cube :: DB:Table •

    View: Dimension + Measure • Slice, Dice & Rotate

  37. Architecture: Revisited 37

  38. Sample Solution

  39. Thank you! Piyush Verma @meson10 Oogway Consulting http://oogway.in