Intro to Parquet (June 2015)

Transcript

1. 1.

   Sam BESSALAH @samklr http://parquet.apache.org

2. 2.

   Typical Data workflow

3. 3.

   Typical Data workflow

4. 4.

   Typical Data workflow

5. 5.

   Typical Data workflow

6. 6.

   Multiple Data Format

7. 7.

   Big Data Data Format Zoo - Sequence Files

8. 8.

   these formats provide

9. 9.

   None
10. 10.

    Binary, columnar storage format for big data analytics workloads, inspired

    by the Google Dremel Paper. - Language independent - Processing framework independent - Formally specified - More than a columnar storage : Dynamic partionning, automatic predicate and projections push down - Awesome performance
11. 11.

    Columnar Storage 101

12. 12.

    Columnar Storage 101

13. 13.

    Columnar Storage 101

14. 14.

    Columnar Storage 101 Advantages : - Limits I/O to the

    data only needed - Big Space savings, better compression, and faster and low overhead encodings - Enables vectorized engine
15. 15.

    Columnar Storage 101

16. 16.

    None
17. 17.

    Parquet Model

18. 18.

    Example Parquet Schema

19. 19.

    None
20. 20.

    None
21. 21.

    Definition and Repetition Levels Definition Level : Stores the level

    for which the field is null Repetition Level : Store levels when new lists are starting in column values.
22. 22.

    None
23. 23.

    None
24. 24.

    None
25. 25.

    None
26. 26.

    None
27. 27.

    None
28. 28.

    Numbers Example: Appnexus 2 MM Logs of Ads impressions 270

    TB of Log Data in Protobuf on HDFS http://techblog.appnexus.com/blog/2015/03/31/parquet-columnar-storage-for-hadoop-data/
29. 29.

    simple bench with HIVE

30. 30.

    None
31. 31.

    None
32. 32.

    Disk Space usage on HDFS with 128 MB blocks

33. 33.

    None
34. 34.

    None
35. 35.

    None
36. 36.

    None
37. 37.

    None
38. 38.

    None
39. 39.

    Slides shamelessly cloned from Julien Le Dem(@J_) , Lead of

    the Apache Parquet Project
40. 40.

    BACKUP SLIDES

41. 41.

    None
42. 42.

    None
43. 43.

    None
44. 44.

    None
45. 45.

    None
46. 46.

    None
47. 47.

    None
48. 48.

    None
49. 49.

    None
50. 50.

    None
51. 51.

    None
52. 52.

    None