A closer look: Exploratory Data Analysis with Spark and IntelliJ IDEA

DataConLA, August 17, 2019
Links:
Zeppelin Notebook: 
https://github.com/MKhalusova/DataConDemo
Sign up for plugin early preview: 
http://bit.ly/BigDataToolsPreview

Transcript

1. A closer look: 
 Exploratory Data Analysis with Spark and IntelliJ IDEA — Maria Khalusova, JetBrains @mariakhalusova
2. !2 About me — • Python • Jupyter • PyCharm • pandas, NumPy • matplotlib, seaborn, etc • scikit-learn, PyTorch, … • Scala • Zeppelin • IntelliJ IDEA • Spark
3. !3 EDA — We all do it Scope may vary: - IDA: assessing data structure, missing values, descriptive statistics, distribution, etc. - Detecting outliers, exploring correlations, discovering patterns, etc. Techniques may vary: - Non-visual - Visualizations: scatterplots, histograms, barplots, etc.
4. !4 EDA — Interactive and iterative Experimental With increasing complexity Ideally using all data
5. !5 Sampling — Pros: fits in memory, we can use pandas Cons: sampling techniques are hard, analysis is less accurate
6. !6 Spark it is then — Ticks all the boxes: (all the data, interactive, experimental) Various APIs Python, Java, Scala
7. !7 Interactivity => Built-in visualization capabilities => As a code editor => Zeppelin — http://zeppelin.apache.org
8. !8 Code editor => Code analysis => Visualizations => Notebooks? IntelliJ IDEA —
9. !9
10. IT’S DEMO TIME —
11. !11 • Data: IntelliJ IDEA repo as a .txt file (1.16 GB) • Spark • Zeppelin • IntelliJ IDEA + BigDataTools + Scala plugin The setup for the EDA (aka Exciting Detective Adventure): —
12. Links: • Zeppelin Notebook: 
 https://github.com/MKhalusova/DataConDemo • Sign up for plugin early preview: 
 http://bit.ly/BigDataToolsPreview @mariakhalusova