Introduction Data-Apps with Python

An introduction to Data-Driven Apps with Python, Pandas and Co. https://github.com/mikebild/introduction-python

Transcript

1. PYTHON DATA-DRIVEN APPS WITH
2. INTRO PYTHON
3. DATA-DRIVEN APPS WITH PYTHON WHY PYTHON? ▸ Simple to Use - Very productive ▸ Easy to Learn ▸ Batteries included (Testing, SQLite Access, CLI-Parse, etc.) ▸ Multiparadigma Language (Procedural, OO, Functional) ▸ Open Source ▸ Awesome Community ▸ Huge Ecosystem and Libraries ▸ Preinstalled on Linux (v3) and OSX (v2)
4. DATA-DRIVEN APPS WITH PYTHON PYTHON2 a, b, *rest = range(10) def f(a, b, *args, option=True): print('Keyword only arguments:', a, b, args, option) f('a', 'b', 'c', option=False) print('Iterators:', ', '.join(map(str, range(3)))) def gen(): yield from ['a', 'b'] print('Yield from:', ', '.join(gen())) UNPACKING MORE IS A ITERATOR KEYWORD ARGUMENTS YIELD FROM PYTHON3 ▸ Support retires in ~2 years! ▸ Some missing language features
5. DATA-DRIVEN APPS WITH PYTHON WHEN PYTHON? ▸ Scripting ▸ Command Line Tools ▸ Web-Development (Flask) ▸ Data-Access (Databases, HTTP/Web, Files, JSON, XML, Excel, etc.) ▸ Data Analytics (NumPy, MatPlotLib, Pandas) ▸ Data Mining / Machine Learning (SciKit-Learn) ▸ GUI-Development (e.g. PyQT for Win, Lin, OSX, Android, iOS)
6. DATA-DRIVEN APPS WITH PYTHON PYTHON CONS? ▸ Not for High Performance - Script based runtime ▸ Design with Types - Python is dynamically typed ▸ Browser Only (Single Page Apps) - Python doesn’t run in it ;-)
7. DATA-DRIVEN APPS WITH PYTHON PYTHON BY A SIMPLE EXAMPLE ▸ Dropbox as Storage ▸ Excel for Data-Manipulation ▸ Flask as Web-Server ▸ Pandas for Data-Analytics ▸ MatPlotLib generates Charts ▸ Docker in Operations (local, prod)
8. BASICS PYTHON
9. DATA-DRIVEN APPS WITH PYTHON TYPES - SIMPLE INTEGER FLOAT COMPLEX STRING BOOLEAN boolean = True and False string = 'Hello World!' integer = 0 float = 0.0 complex = 0j NONE none = None INPUT / OUTPUT input = input("Input please: ") print('Output:', input)
10. DATA-DRIVEN APPS WITH PYTHON TYPES - SEQUENCES LIST TUPLE DICTIONARY SET list = ['a', 1, 6.2] tuple = (1, 'a', 0.0) dictionary = { 'key1': 'value 1', 'key2': 'value 2' } immutable_set = set(['a', 'b']) ITERATION for item in list: # continue # break print(item)
11. DATA-DRIVEN APPS WITH PYTHON BASICS - CONDITIONS if list: print('Is a list') elif not list: print('Is not a list') else: print('What else ;-)') ▸ Integer - 0 ▸ Long - 0L ▸ Float - 0.0 ▸ Complex - 0j ▸ Boolean - False ▸ List - [] ▸ Tuple - () ▸ Dictionary - {} ▸ None - None FALSE something = '123' if list else '456' SHORT
12. DATA-DRIVEN APPS WITH PYTHON FUNCTIONS def my_function(name='Mike'): """A 'Hello World!' function""" return 'Hello ' + name print("Output:", my_function('John')) my_func = lambda name='Mike': 'Hello ' + name print("Output:", my_func('John')) LAMBDAS DOCSTRINGS help(my_function.__doc__) ▸ Functions are Objects with Functions and Properties ▸ Variable Function Arguments (*as_tuple, **as_dictionary) ▸ NO Tail Recursion Optimization
13. DATA-DRIVEN APPS WITH PYTHON CLASSES class Parent(object): value = 0 __private_value = 0 def __init__(self, value = 100): self.__private_value = value self.value = value + value def getInstance(self): return self def getPrivateValue(self): return self.__private_value parent = Parent(10) print(dir(parent)) # print('Class:', parent.value, parent.getPrivateValue()) ▸ Magic-Functions (__str__, __cmp__, etc.) in Classes ▸ Built-In Class Attributes (__name__, __doc__, etc.) ▸ __foo means Private Function
14. DATA-DRIVEN APPS WITH PYTHON MODULES import sys print('Module Search Path:', sys.path) from sys import path print('Module Search Path:', path) import my_module import my_module as mm from my_module import my_func """ The sample module (my_module.py) """ def my_func(name='Mike'): """A 'Hello World!' function""" return 'Hello ' + name
15. DATA-DRIVEN APPS WITH PYTHON TESTING import unittest import testing_sut class MyTestClass(unittest.TestCase): @classmethod def setUpClass(cls): pass @classmethod def tearDownClass(cls): pass def setUp(self): pass def tearDown(self): pass def test_answer_should_success(self): assert testing_sut.func(3) == 4 def test_answer_should_fail(self): assert testing_sut.func(3) == 5 python -m unittest testing.py python testing.py
16. DATA-DRIVEN APPS WITH PYTHON SOME MORE PYTHON… ▸ Strings ▸ Runtime, OS and Environment ▸ Regular Expressions ▸ Curses (Text-UI) ▸ Async IO (coroutines, async/await, eventloop, etc.)
17. DATA-DRIVEN APPS WITH PYTHON SOME MORE PYTHON SPECIALS… ▸ Python Image Library ▸ Database Development (SQLite, MySQL, …) ▸ UI Development (PyQt)
18. CLI- DEVELOPMENT PYTHON
19. DATA-DRIVEN APPS WITH PYTHON SCRIPTING INTERPRETER #!/usr/bin/env python3 print('Hello World!') python app.py SCRIPT ./app.py chmod +x app.py
20. DATA-DRIVEN APPS WITH PYTHON DOC GENERATION (HTML) #!/usr/bin/env python3 """ The sample module """ def main(name='Mike'): """A 'Hello World!' function""" return 'Hello ' + name if __name__ == "__main__": print(main()) pydoc -w ./app.py chmod +x app.py
21. DATA-DRIVEN APPS WITH PYTHON CLI ARGUMENT PARSING import argparse parser = argparse.ArgumentParser() parser.add_argument("-v", "--verbosity", action="store_true", default=0, help="increase output verbosity") parser.add_argument("-p", "--port", default=8080, help="set port") args = parser.parse_args() print('Verbosity level:', args.verbosity) print('Port:', args.port) python my_cli.py -v -p 9090 python my_cli.py -h
22. WEB- DEVELOPMENT PYTHON
23. DATA-DRIVEN APPS WITH PYTHON FLASK IS ▸ Open-Source Micro-Framework for Web-Development ▸ Easy to use and configure ▸ Excellent documentation ▸ RESTful WebServices ▸ Server-Side Rending HTML Jinja - http://jinja.pocoo.org ▸ Testable
24. DATA-DRIVEN APPS WITH PYTHON KEEP CODE SIMPLE - FLASK EXTENSIONS ▸ Flask-Admin ▸ Flask-Restful (https://flask-restful.readthedocs.io/en/latest/) ▸ Flask-Cache ▸ Flask-Mail ▸ Flask-Testing ▸ more, more, more …
25. DATA-DRIVEN APPS WITH PYTHON LET’S CODE - FIRST APP INSTALL CODING RUN from flask import Flask app = Flask(__name__) @app.route('/') def hello_world(): return 'Hello World!' if __name__ == '__main__': app.run(debug=True, port=8080) pip install Flask python 1-app.py FLASK_APP=1-app.py FLASK_DEBUG=1 flask run -p 8080 -h 0.0.0.0
26. DATA-DRIVEN APPS WITH PYTHON ROUTING - URL - QUERY PARAMETERS from flask import Flask, request @app.route('/users', defaults={'username': None}) @app.route('/users/<username>') def show_user_profile(username=None): fullname = request.args.get('fullname') return 'Username: %s (%s)' % (username, fullname) STATIC FILES (DEFAULT) HTTP METHODS / STATUS-CODE REDIRECTS @app.route('/', methods=['GET', 'POST']) def hello_world(): return 'Hello World!’, 202 app = Flask(__name__, static_url_path='/static') from flask import Flask, redirect, url_for @app.route('/login') def login(): return redirect(url_for('hello_world'))
27. DATA-DRIVEN APPS WITH PYTHON JSON REQUEST/RESPONSE from flask import Flask, jsonify, request app = Flask(__name__) @app.route('/json') def get_username(): return jsonify(username='John Doe') app.run(debug=True, port=8080) curl http://localhost:8080/json -H 'accept: application/json' GET JSON
28. DATA-DRIVEN APPS WITH PYTHON JSON REQUEST/RESPONSE from flask import Flask, jsonify, request app = Flask(__name__) @app.route('/json', methods=['GET', 'POST']) def get_username(): print('Request:', request.json) res = { 'username': 'John Doe', } if request.json: res.update(request.json) else: res return jsonify(res) if __name__ == '__main__': app.run(debug=True, port=8080) curl -XPOST http://localhost:8080/json -H 'content-type: application/json' -d '{"foo": "bar"}' POST JSON
29. DATA-DRIVEN APPS WITH PYTHON HTML-TEMPLATES from flask import Flask, request, render_template app = Flask(__name__) @app.route('/') def hello(): return render_template('2-html.html') @app.route('/forms', methods=['POST']) def forms(): return 'You said: ' + request.form['text'] if __name__ == '__main__': app.run(debug=True, port=8080) <form action="/forms" method="POST"> <input name="text"> <input type="submit" value="Submit"> </form> {% if text %} You said: {{ text }} {% endif %} With Jinja - http://jinja.pocoo.org templates/2-html.html
30. DATA-DRIVEN APPS WITH PYTHON HANDLE LARGER DATA - STREAM from flask import Flask, stream_with_context, request, Response @app.route(‚/stream‘) def streamed_response(): @stream_with_context def generate(): yield 'Hello;' yield request.args['name'] return Response(generate(), mimetype='text/csv')
31. DATA-DRIVEN APPS WITH PYTHON TESTING - FLASK-TEST-CLIENT from app import app import unittest class FlaskAppTests(unittest.TestCase): def setUp(self): self.app = app.test_client() self.app.testing = True def test_home_status_code(self): result = self.app.get('/') self.assertEqual(result.status_code, 202) def test_home_data(self): result = self.app.get('/') self.assertEqual( b'Hello World!', result.data) if __name__ == "__main__": unittest.main() python -m unittest 5-testing.py python 5-testing.py
32. DATA-ACCESS PYTHON
33. DATA-DRIVEN APPS WITH PYTHON ACCESS ▸ Files ▸ Databases ▸ HTTP-API Client FORMATS ▸ JSON ▸ XML ▸ CSV ▸ Excel
34. DATA- ANALYTICS PYTHON
35. DATA-DRIVEN APPS WITH PYTHON BIG DATA? OVERVIEW ▸ Access - sources, size and performance ▸ Quality - un/structured and consistency ▸ Processing - fusion, filter, transformation, aggregation, grouping of data ▸ Visualization - charts, graphs and other displays of the data
36. DATA-DRIVEN APPS WITH PYTHON OVERLAPS WITH ▸ Business Intelligence ▸ Data Collection and Consolidation ▸ Data Standardization, Cleanup and Profiling ▸ Modeling of Information Architecture ▸ Statistical Analytics and Pattern-Recognition ▸ Data Science ▸ Data/Text Mining - Statistical techniques to find cross-connection, content and trends ▸ Machine Learning - un-/semi-/supervised algorithms for data analytics and mining ▸ Smart Data ▸ Prepared within Business-, User-, Contextual Relation ▸ Highly Valuable & High Quality ▸ Inclusion of Data Security and Protection
37. DATA-DRIVEN APPS WITH PYTHON SOME POPULAR LIBRARIES ▸ NumPy - Numerical function & High Performance Vector + Matrix computations ▸ MatPlotLib - Data visualization ▸ Pandas - Easy-to-use data structures for data analysis ▸ SciKit-Learn - un/supervised/semi-supervised algorithms
38. DATA-DRIVEN APPS WITH PYTHON NUMPY ▸ High performance multidimensional array object ▸ Has typed dimensions (vs. lists) ▸ Tools for working with arrays and matrices ▸ Load / Save data pip install numpy INSTALL
39. DATA-DRIVEN APPS WITH PYTHON NUMPY python 1-numpy.py import numpy as np # Creation a = np.array([1, 2, 3]) print('Creation:', a)
40. DATA-DRIVEN APPS WITH PYTHON MATPLOTLIB ▸ 2D and 3D scientific plots ▸ Fine-grained control over every aspect ▸ Many output file formats including PNG, PDF, SVG, EPS pip install matplotlib INSTALL
41. DATA-DRIVEN APPS WITH PYTHON MATPLOTLIB import pylab as pl pl.ioff() pl.isinteractive() x = [1 ,3 ,7] pl.plot(x) # Save as PDF pl.savefig('fig_test.pdf', dpi=600, format='pdf') # Show pl.show() python 1-matplotlib.py
42. DATA-DRIVEN APPS WITH PYTHON PANDAS ▸ Data Sources / Destinations ▸ Data Structures - DataFrame, DataSeries ▸ Filter / Transform / Aggregate / Order ▸ Grouping ▸ Pivot ▸ Visualization pip install pandas INSTALL
43. DATA-DRIVEN APPS WITH PYTHON PANDAS import pandas as pd # Series series = pd.Series([1, 2.1, 3], index=['a', 'b', 'c'], dtype=int) print('Series:', series) # DataFrame data = { 'Country': ['Germany', 'India', 'Swiss'], 'Capital': ['Berlin', 'New Delhi', 'Bern'], 'Population': [82457000, 1339180000, 8401000, ] } df = pd.DataFrame(data, columns=['Country', 'Capital', 'Population']) print('DataFrame:', df) python 1-pandas.py
44. DEPLOYMENT & OPERATION PYTHON
45. DATA-DRIVEN APPS WITH PYTHON ENVIRONMENT VARIABLES #!/usr/bin/env python3 import os print('Home:', os.environ['HOME']) print('My:', os.environ.get('MY')) print('Foo:', os.environ.get('FOO', 'default_value')) FOO=bar ./env-os.py
46. DATA-DRIVEN APPS WITH PYTHON UNIX SIGNAL HANDLING #!/usr/bin/env python3 import signal import sys def signal_handler(signal, frame): print('You pressed Ctrl+C!') sys.exit(0) signal.signal(signal.SIGINT, signal_handler) print('Press Ctrl+C') signal.pause() chmod +x signal-handler.py ./signal-handler.py
47. DATA-DRIVEN APPS WITH PYTHON PIP - PYTHON PACKAGE MANAGER pip freeze > requirement.txt pip install -r requirement.txt pip install some-package-name pip uninstall some-package-name DEPENDENCY FILE
48. DATA-DRIVEN APPS WITH PYTHON DOCKER FROM python:3-alpine RUN apk --no-cache --update-cache add gcc gfortran python python-dev py-pip build-base wget freetype-dev libpng-dev ADD . /app WORKDIR /app RUN pip install --no-cache-dir -r requirements.txt EXPOSE 80 CMD ["python", "server.py"] docker run my_python_app docker build -t my_python_app .