Data Analysis Project in Python

Outline: So you have some data, you can do Python. Now how do you go about organizing a data analysis project? What software design approach to use? Which libraries are available in Python? What are the common pitfalls and how to avoid them?

Transcript

1. Data Analysis Project in Python do’s and don’ts Oxana Sachenkova | 2013-09-17 | for PyLadies Stockholm
2. About me
3. Data Analysis Deriving a valuable insights from data in… ★ Finance ★ Marketing and Sales ★ Natural Language Processing ★ Science ◦ Biology ◦ Physics ◦ Chemistry …..
4. • Descriptive • Exploratory • Inferential • Predictive • Causal • Mechanistic Data Analysis
5. “I just need to run it once...” Software vs Results
6. Big data Reproducible research Open science Why is it important?
7. Step by step... Formulate the question Collect data Process data Descriptive analysis Formal analysis Visualization Tidy data - relational databases in the light of statistics http://vita.had.co. nz/papers/tidy-data.pdf
8. Project structure experiment1_files/ analysis.py fg1.R sequences.aln sequences_.aln experiment2_files/ sequences_.aln analysis_hmg.py strands.avg.dat
9. Structured programming Dijkstra, 1968 Software architecture and design Methodologies
10. Project structure principles ➔ Logically separated functional modules ➔ Meaningful names ➔ Easy to reconfigure ➔ Reuse code and data ➔ Structure your results
11. Project skeleton example On github: https://github. com/OxanaSachenkova/data- analysis-sample For a general app: https://github. com/kennethreitz/samplemod
12. Python packages import towelstuff as towel towel.get_location() http://guide.python-distribute.org/
13. What if I don’t know? ….what comes next...
15. Agile data analysis ➔ work in small iterations with frequent feedback and course correction ➔ verify your data on every “release”; turn bugs into test cases ➔ use version control ➔ collaborate
16. Step by step... Formulate the question Collect data Process data Descriptive analysis Formal analysis Visualization
17. IPython Quick guide to IPython
18. Structured code Code and Pasta Spagetti Ravioli Lasagna What to do? Don’t repeat yourself Keep it simple Refactoring Pair Programming
19. Inspiration https://github.com/pydata/pandas/blob/master/pandas/computation/expressions.py
20. Structured code Be elegant PEP8 Google Coding Style Be pythonic Use the Force Evolution of a Python programmer
21. Example: strings and lists
22. When in doubt... $ python >>> import this Now is better than never
23. I haven’t covered How to actually perform data analysis in Python :-) How to write tests for data analysis software? Test driven development for data analysis Code profiling and optimization (vbench) Automatically generated documentation
24. NumPy/Scipy Have a look at the Example List and John Cook's Distributions in Scipy . pandas Working with tabular data like a boss larry Labeled array that plays nice with NumPy. python-statlib Several statistical libraries in one. statsmodels Statistical modeling: Linear models, GLMs. scikits Machine learning. matplotlib Data visualization Python libraries for data analysis
25. Data Analysis resources Wes McKinney’s tutorials 1 and 2 on Kaggle. Hernan Rojas’ tutorial Tutorials on financial data and time series using pandas Python for Data Analysis (book) Interesting IPython Notebooks
26. More on best practices The Hitchhiker's Guide to Python Python worst practices Quick guide to organizing Computational Biology projects Best practices for Scientific Computing
27. PyLadies Stockholm Let us know what you think Share your ideas at the next brainstorming session Suggest the next meetup topic online Tell your friends @PyLadiesSthlm
28. Thank you!