Slide 1
Slide 1 text
Snakes in a Buildpack
By Ian Huston and James Wen
Slide 2
Slide 2 text
Ian has a problem
Hard to deploy apps using great Python data science packages
Slide 3
Slide 3 text
Why is this important?
#productsNOTpowerpoints
Slide 4
Slide 4 text
Current Solution
Community Buildpack:
https://github.com/ihuston/python-conda-buildpack
Not tested
No CI pipeline
No time for maintenance
Slide 5
Slide 5 text
Other Attempted Solutions
Kenneth Reitz Buildpack
● https://github.com/kennethreitz/conda-buildpack
● Looks unmaintained, Heroku-specific, didn’t
work with CF
Continuum’s Conda buildpack
● https://github.com/conda/conda-buildpack
● No updates for 2 years, Heroku specific,
would need downstream changes
Just use Pip!
● https://jakevdp.github.io/blog/2016/08/25/co
nda-myths-and-misconceptions/
Docker?
● Don’t want responsibility of maintaining OS
layer!
Slide 6
Slide 6 text
Containers vs Buildpacks
runtime layer
OS image
application layer
Container (e.g. Docker)
system brings
fixed host OS
Kernel
* Devs may bring a custom
buildpack
runtime layer*
OS image
application layer
Buildpack
App container
System Provides
Dev Provides
system brings
fixed host OS
Kernel
Slide 7
Slide 7 text
Experiments
- Use miniconda in the CF python buildpack
- Vendor anaconda
- Add data science dependencies to the rootfs
Solution: Port over Ian’s conda buildpack and maintain it as a code
path in the current python buildpack.
Slide 8
Slide 8 text
Improvement/Fixes needed after
Lock miniconda to a version instead of
downloading latest
Miniconda was always installing python 3
Miniconda needed to save and load from
natural app cache
Suppress miniconda progress bar output
(massive staging logs)
Slide 9
Slide 9 text
What can we do now?
Predict time to delivery for an
international courier company
Order the right components
for an European car
manufacturer
Deliver warnings of
dangerous road conditions
while you drive
Slide 10
Slide 10 text
Hobbyist: OMSCS Machine Learning Course
- Taking a Machine Learning class for the GTech OMSCS program
- HW Assignment = Applying 5 supervised machine learning models to 2 large data sets
Slide 11
Slide 11 text
Hobbyist: OMSCS Machine Learning Course
- For the experiments/analyses, using Pivotal Web Services > running on my dinky 4 year old macbook
- Ran the experiments by wrapping them in minimal Flask apps
Slide 12
Slide 12 text
Future of Conda in the Python Buildpack
Your Feedback?
- Can’t run this functionality in air-gapped environment
- Conda cannot vendor your packages (i.e. something
like `bundle package`)
- Reduce size of end droplet?
- Apps using conda often end up large due to slew of
dependencies
Vendor miniconda (cached buildpack)
Slide 13
Slide 13 text
Where can I get it?
Official CF Python Buildpack from v1.5.6:
https://github.com/cloudfoundry/python-buildpack
Slide 14
Slide 14 text
Getting your wonderful ideas into Cloud Foundry
- Open up discussion on cf-dev mailing list - Open up discussion on open source Cloud Foundry slack
- Do the CF Dojo program and gain the skills to
work on CF full-time (or just join Pivotal)
- Create a feature narrative and get feedback
Slide 15
Slide 15 text
Any Questions?
James Wen @rochesterinnyc
Ian Huston @ianhuston
https://github.com/cloudfoundry/python-buildpack