Python for Data Science at Pivotal

Python for Data
Science at Pivotal
•  Ian Huston, Data Scientist
Ian Huston, Data Scientist
Python Ireland August 2015

View Slide

2
© Copyright 2015 Pivotal. All rights reserved.
Who am I?
  Ian Huston
  @ianhuston
  www.ianhuston.net
  Data Scientist
  Use PyData stack for
predictive analytics and
machine learning
  Previously a theoretical
physicist using Python for
numerical simulations & HPC

View Slide

3
Who are Pivotal?
OPEN DATA
PLATFORM
Pivotal
Big Data Suite

View Slide

4
NOW HIRING IN DUBLIN!

View Slide

Data Scientist (n.):

Person who is better at statistics than any
software engineer and better at software
engineering than any statistician.

- Josh Wills

View Slide

6
Plan
1  Python for Data packages and tools
2  Python in your database
3  Python in the cloud

View Slide

View Slide

8
Why Python?
  Powerful & simple syntax – great for interactive work
  Backed up with fast C & Fortran numerical libraries
  Growing community and set of libraries
  R is still extremely popular in data science
  GIL and multi-core support make scaling Python diﬃcult

View Slide

1.  PyData packages !
and tools

View Slide

10
Python for Data community
  Pycon Ireland Data Science track!
  PyData conferences + videos
–  London, Berlin, multiple US locations
  #PyData on Twitter

View Slide

11
Packages - Data Manipulation
  Low level array operations
  Data tables and in-memory manipulation
  Parallel out-of-core array manipulation
  High level interface for databases and
diﬀerent computational backends
NumPy
Dask

View Slide

12
Packages - Modelling
  FFTs, integration, other general algorithms
  Statistical distributions and tests
  Machine Learning pipelines
  Bayesian Probabilistic Programming
SciPy
PyMC3

View Slide

13
Packages - Visualisation
  Widely used and powerful plotting package
  Opinionated but beautiful data visualisations
  Interactive plotting with server option
  Graphics API with translation between
languages (e.g. Python -> D3)
seaborn
Bokeh

View Slide

14
IPython Notebooks
http://nbviewer.ipython.org/gist/fonnesbeck/2352771

View Slide

15

View Slide

16
PREDICT THE
DESTINATION

View Slide

17
PREDICT THE
RANGE

View Slide

18
Connected Car
http://tinyurl.com/pivotal-car
https://github.com/pivotal/IoT-ConnectedCar

View Slide

2.  In-Database Python!
(and R, Java, C, etc)

View Slide

20
Bring your code to the data
  Procedural Python – support in PostgreSQL + others
  Use the expressive power of Python inside the database
  Reduce/remove large data movements
  Couple with distributed databases for simple parallelisation

View Slide

CREATE FUNCTION
pymax (a integer, b integer)
RETURNS integer
AS $$
if a > b:
return a
return b
$$ LANGUAGE plpythonu;

SQL wrapper
Language
Normal Python

View Slide

22
Data Parallelism
  Little or no eﬀort is required to break up the problem into a
number of parallel tasks, and there exists no dependency
(or communication) between those parallel tasks.
  Examples:
–  Measure the height of each student in a classroom (explicitly
parallelizable by student)
–  MapReduce
–  map() function in Python

View Slide

23
PostgreSQL
PostgreSQL PostgreSQL PostgreSQL PostgreSQL

View Slide

PostgreSQL

View Slide

BENEFITS:
Reuse existing Python code
Access Python libraries
Implicit parallelism

View Slide

26
Natural Language Processing in-database
  Business Problem:  
Want to understand what is being discussed in millions of
documents and whether authors feel positive about us
  Topic Modelling: 
Characterise documents based on topics contained within
  Sentiment Analysis: 
Score documents based on ‘sentiment’ (positive or negative)
Natural Language ToolKit (NLTK)

View Slide

27
Topic and Sentiment Analysis Pipeline
Documents
Load into
database
Parallel
Parsing of
JSON using
PL/Python
Topic
Modelling
Sentiment
Analysis
D3.js
http://vimeo.com/79558274
NLTK

View Slide

3.  Python in the Cloud

View Slide

What do data scientists need?

View Slide

Cloud Applications Haiku

Here is my source code
Run it on the cloud for me
I do not care how.
-  Onsi Fakhouri
@onsijoe

View Slide

What is Cloud Foundry?
http://cloudfoundry.org

Open Source
Multi-Cloud Platform

Simple App Deployment,
Scaling & Availability

View Slide

$ cf push

View Slide

33
Simple Flask App Demo
  Simple one page “Hello World” web app
  Video: https://www.youtube.com/watch?v=QOfD6tnoAB8
  Demonstrates:
–  Installation of requirements
–  Scaling properties
  Need to Provide:
–  App files
–  Dependencies listed in requirements.txt file
–  Optional manifest.yml file with configuration for deployment

View Slide

C
F

R
O!
U!
T!
E!
R
2. Set up domain
Cloud
Controller
Instance
1. Upload code
4. Copy app into
containerised
instances
3. Install Python
&
Dependencies
5. Start app
and accept
connections
Send request to URL
WHAT JUST
HAPPENED?
Source
Code
Instance
$ cf push
Browser
5. Load balance
between
instances

View Slide

35
Python on Cloud Foundry
  First class language (with Go, Java, Ruby, Node.js, PHP)
  Automatic app type detection
–  Looks for requirements.txt or setup.py
  Buildpack takes care of
–  Detecting that a Python app is being pushed
–  Installing Python interpreter
–  Installing packages in requirements.txt using pip
–  Starting web app as requested (e.g. python myapp.py)

View Slide

36
Official Python Buildpack
ü  Great for simple pip based requirements
ü  Well tested and officially maintained
ü  Covers both Python 2 and 3
✗ Suffers from the Python Packaging Problem:
-  Hard to build packages with C, C++ or Fortran extensions
-  Complicated local configuration of libraries and paths needed
-  Takes a long time to build main PyData packages from source

View Slide

37
Using conda for package management
  http://conda.pydata.org
  Beneﬁts:
–  Uses precompiled binary packages
–  No ﬁddling with Fortran or C compilers and library paths
–  Known good combinations of main package versions
–  Really simple environment management (better than virtualenv)
–  Easy to run Python 2 and 3 side-by-side
Go try it out if you haven’t already!

View Slide

38
How to use the conda buildpack
https://github.com/ihuston/python-conda-buildpack
  Specify as a custom buildpack when pushing app with
manifest or -‐b command line option.
  Export your current environment to a environment.yml ﬁle
  Or write requirements.txt (pip) and conda_requirements.txt
  Send me feedback & pull requests!

View Slide

R
E
S
T

A
P
I
Send data as JSON
Data
Ingest
Model
Create Model
Redis
Kicking off
periodic
retraining
Save training
data
Save model
object
Send JSON data
without label
Receive prediction
from trained model
instance
Deployed at:!
http://dsoncf.cfapps.io!
Code:
https://github.com/pivotalsoftware/ds-cfpylearning
PREDICTION API
ARCHITECTURE
$ cf create-‐service
rediscloud
PLAN_NAME
INSTANCE_NAME

View Slide

TRANSPORT
DISRUPTION!
PREDICTIONS
http://ds-demo-transport.cfapps.io

View Slide

41

Show off your data
science related Cloud
Foundry apps:

Twitter: @dsoncf
http://dsoncf.com

View Slide

42
Resources
  PyData.org
  PL/Python – see PostgreSQL docs
  CloudFoundry.org
We’re hiring in Dublin & London:
pivotal.io/careers Kevin Olsen [email protected]

View Slide

43
@ianhuston

View Slide

44
Appendix

View Slide

C
F

R
O!
U!
T!
E!
R
2. Set up domain
Cloud
Controller
Instance
1. Upload code
4. Copy app into
containerised
instances
3. Install Python
&
Dependencies
5. Start app
and accept
connections
Send request to URL
WHAT JUST
HAPPENED?
Source
Code
Instance
$ cf push
Browser
5. Load balance
between
instances

View Slide

46
What just happened?
1.  Application code is uploaded to CF
2.  Domain URL is set up ready for routing
3.  Cloud controller builds application in container:
–  Python interpreter selected
–  Dependencies installed with pip
4.  Container is replicated to provide instances
5.  App starts and Router load balances requests

View Slide

47
Containers vs Buildpacks
runtime layer
OS image
application layer
Container (e.g. Docker)
system brings ﬁxed
host OS Kernel
* Devs may bring a custom
buildpack
runtime layer*
OS image
application layer
Buildpack
App container
System Provides
Dev Provides
system brings ﬁxed
host OS Kernel

View Slide

Python for Data Science at Pivotal

Python for Data Science at Pivotal

Ian Huston

More Decks by Ian Huston

Other Decks in Technology

Featured

Transcript