Getting Started with Cloud Foundry

Getting Started with Cloud Foundry

Talk given at ODSC London meetup on March 16th 2016

As a data scientist I frequently need to create web apps to provide interactive functionality, deliver data APIs or simply publish results. It is now easier than ever to deploy your data driven web app by using cloud based application platforms to do the heavy lifting. In this talk I will introduce Cloud Foundry (http://cloudfoundry.org) an open source cloud platform that enables simple app deployment, scaling and connectivity to data services like PostgreSQL, MongoDB, Redis and Cassandra.

41d2c569bbfbec97e0ab6fd2a8c261b7?s=128

Ian Huston

March 16, 2016
Tweet

Transcript

  1. Getting Started with Cloud Foundry •  Ian Huston, Data Scientist

    Ian Huston, Data Scientist ODSC London 2016
  2. 2 © Copyright 2015 Pivotal. All rights reserved. Who am

    I? Ÿ  Ian Huston Ÿ  @ianhuston Ÿ  www.ianhuston.net Ÿ  More resources:
 https://github.com/ihuston/ python-cf-examples Ÿ  Data Scientist Ÿ  Previously a theoretical physicist using Python for numerical simulations & HPC
  3. 3 © Copyright 2015 Pivotal. All rights reserved. Aims for

    this talk Goals: Ÿ  What is Cloud Foundry and how can it help? Ÿ  Get a taste of CF as a developer/data scientist Anti-goals: Ÿ  Understand CF from an operations viewpoint Ÿ  Install & run a full Cloud Foundry installation
  4. 4 © Copyright 2015 Pivotal. All rights reserved. Talk Resources

    For the full 2 hour tutorial clone this repo: https://github.com/ihuston/python-cf-examples

  5. What is Cloud Foundry? http://cloudfoundry.org Open Source Multi-Cloud Platform Simple

    App Deployment, Scaling & Availability
  6. Cloud Applications Haiku Here is my source code Run it

    on the cloud for me I do not care how. -  Onsi Fakhouri @onsijoe
  7. How can data scientists use CF?

  8. Data Services Easy control of incoming data

  9. Distributed computation

  10. Data Driven Applications

  11. 11 © Copyright 2015 Pivotal. All rights reserved. Multi-cloud Ÿ 

    Same applications running across different cloud providers: Private Public Hosted
  12. 12 © Copyright 2015 Pivotal. All rights reserved. Cloud Foundry

    Foundation PLATINUM GOLD SILVER
  13. 13 © Copyright 2015 Pivotal. All rights reserved. cf push

    Pushing your first app
  14. 14 © Copyright 2015 Pivotal. All rights reserved. Challenge 1:

    Pushing your first application Ÿ  Choose PWS API endpoint: $ cf api https://api.run.pivotal.io Ÿ  Login: $ cf login Ÿ  Code directory: 01-simple-python-app Ÿ  Deploy with $ cf push Ÿ  Check it worked at http://myapp-RANDOM-WORDS.cfapps.io Ÿ  Turn off your app when finished with cf stop myapp (but not yet!)
  15. 15 © Copyright 2015 Pivotal. All rights reserved. Simple Flask

    App Demo Ÿ  Simple one page “Hello World” web app Ÿ  Video: https://www.youtube.com/watch?v=QOfD6tnoAB8 Ÿ  Demonstrates: –  Installation of requirements –  Scaling properties Ÿ  Need to Provide: –  App files –  Dependencies listed in requirements.txt file –  Optional manifest.yml file with configuration for deployment
  16. C F R O" U" T" E" R 2. Set

    up domain Cloud Controller Instance 1. Upload code 4. Copy app into containerised instances 3. Install Python & Dependencies 5. Start app and accept connections Send request to URL WHAT JUST HAPPENED? Source Code Instance $ cf push Browser 5. Load balance between instances
  17. 17 © Copyright 2015 Pivotal. All rights reserved. What just

    happened? 1.  Application code is uploaded to CF 2.  Domain URL is set up ready for routing 3.  Cloud controller builds application in container: –  Python interpreter selected –  Dependencies installed with pip 4.  Container is replicated to provide instances 5.  App starts and Router load balances requests Ÿ  See what’s happening using logs: $ cf logs myapp --recent
  18. 18 © Copyright 2015 Pivotal. All rights reserved. Python on

    Cloud Foundry Ÿ  First class language (with Go, Java, Ruby, Node.js, PHP) Ÿ  Automatic app type detection –  Looks for requirements.txt or setup.py Ÿ  Buildpack takes care of –  Detecting that a Python app is being pushed –  Installing Python interpreter –  Installing packages in requirements.txt using pip –  Starting web app as requested (e.g. python myapp.py)
  19. 19 © Copyright 2015 Pivotal. All rights reserved. Scaling memory

    and instances Ÿ  We can scale our application as needed in terms of memory and number of instances: $ cf scale myapp –i 5 $ cf scale myapp –m 256M Ÿ  Check app in browser to see different instances being used. Ÿ  Scale back down with $ cf scale myapp –i 1
  20. 20 © Copyright 2015 Pivotal. All rights reserved. How does

    this work? Ÿ  Containerised application is cached and ready to be deployed. Ÿ  Scaling number of instances replicates container and load balances requests across all instances. Ÿ  Scaling memory requires restarting app. Ÿ  Auto-scaling is also possible.
  21. 21 © Copyright 2015 Pivotal. All rights reserved. Buildpacks Pushing

    data science apps
  22. 22 © Copyright 2015 Pivotal. All rights reserved. What are

    buildpacks? Ÿ  Idea and format from Heroku Ÿ  Responsible for doing whatever is necessary to get your app running. Ÿ  Buildpacks take care of –  Detecting which type of application is being pushed –  Installing the appropriate run-time –  Installing required dependencies or other artifacts –  Starting the application as requested
  23. 23 © Copyright 2015 Pivotal. All rights reserved. Containers vs

    Buildpacks runtime layer OS image application layer Container (e.g. Docker) system brings fixed host OS Kernel * Devs may bring a custom buildpack runtime layer* OS image application layer Buildpack App container System Provides Dev Provides system brings fixed host OS Kernel
  24. 24 © Copyright 2015 Pivotal. All rights reserved. Community Buildpacks

    https://github.com/cloudfoundry-community/ cf-docs-contrib/wiki/Buildpacks Ÿ  Lots of languages: 
 Clojure, Haskell, .NET, Erlang, Elixir, etc. Ÿ  RShiny app buildpack: 
 https://github.com/alexkago/cf-buildpack-r Ÿ  Can also use some Heroku buildpacks without modification.
  25. 25 © Copyright 2015 Pivotal. All rights reserved. Using conda

    for package management Ÿ  http://conda.pydata.org Ÿ  Benefits: –  Uses precompiled binary packages –  No fiddling with Fortran or C compilers and library paths –  Known good combinations of main package versions –  Really simple environment management (better than virtualenv) –  Easy to run Python 2 and 3 side-by-side Go try it out if you haven’t already!
  26. 26 © Copyright 2015 Pivotal. All rights reserved. Challenge 2:

    Pushing a PyData app Ÿ  Code directory: 02-pydata-spyre-app Ÿ  Spyre – Adam Hajari https://github.com/adamhajari/spyre Ÿ  This app uses the Pydata buildpack to install Matplotlib, NumPy and more. Ÿ  Spyre provides a simple way to build interactive web based visualisations similar to Rshiny.
  27. 27 © Copyright 2015 Pivotal. All rights reserved. Next Steps

    See https://github.com/ihuston/python-cf-examples for continuation of tutorial: Ÿ  How to bind data sources like Redis & PostgreSQL Ÿ  Build a simple machine learning API system Full tutorial as a video: https://www.youtube.com/watch?v=lp2c05yDJko
  28. 28 © Copyright 2015 Pivotal. All rights reserved. Resources Ÿ 

    Docs: http://docs.cloudfoundry.org Ÿ  CF Summit videos: http://cfsummit.com Ÿ  Join the CF community: http://cloudfoundry.org Ÿ  CF meetups: http://cloud-foundry.meetup.com
  29. 29 © Copyright 2015 Pivotal. All rights reserved. @ianhuston