Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Getting Started with Cloud Foundry

Getting Started with Cloud Foundry

Talk given at ODSC London meetup on March 16th 2016

As a data scientist I frequently need to create web apps to provide interactive functionality, deliver data APIs or simply publish results. It is now easier than ever to deploy your data driven web app by using cloud based application platforms to do the heavy lifting. In this talk I will introduce Cloud Foundry (http://cloudfoundry.org) an open source cloud platform that enables simple app deployment, scaling and connectivity to data services like PostgreSQL, MongoDB, Redis and Cassandra.

Ian Huston

March 16, 2016
Tweet

More Decks by Ian Huston

Other Decks in Technology

Transcript

  1. Getting Started with
    Cloud Foundry
    •  Ian Huston, Data Scientist
    Ian Huston, Data Scientist
    ODSC London 2016

    View Slide

  2. 2
    © Copyright 2015 Pivotal. All rights reserved.
    Who am I?
    Ÿ  Ian Huston
    Ÿ  @ianhuston
    Ÿ  www.ianhuston.net
    Ÿ  More resources:

    https://github.com/ihuston/
    python-cf-examples
    Ÿ  Data Scientist
    Ÿ  Previously a theoretical
    physicist using Python for
    numerical simulations & HPC

    View Slide

  3. 3
    © Copyright 2015 Pivotal. All rights reserved.
    Aims for this talk
    Goals:
    Ÿ  What is Cloud Foundry and how can it help?
    Ÿ  Get a taste of CF as a developer/data scientist
    Anti-goals:
    Ÿ  Understand CF from an operations viewpoint
    Ÿ  Install & run a full Cloud Foundry installation

    View Slide

  4. 4
    © Copyright 2015 Pivotal. All rights reserved.
    Talk Resources
    For the full 2 hour tutorial clone this repo:
    https://github.com/ihuston/python-cf-examples



    View Slide

  5. What is Cloud Foundry?
    http://cloudfoundry.org

    Open Source
    Multi-Cloud Platform

    Simple App Deployment,
    Scaling & Availability

    View Slide

  6. Cloud Applications Haiku

    Here is my source code
    Run it on the cloud for me
    I do not care how.
    -  Onsi Fakhouri
    @onsijoe

    View Slide

  7. How can data scientists use CF?

    View Slide

  8. Data Services
    Easy control of incoming data

    View Slide

  9. Distributed computation

    View Slide

  10. Data Driven Applications

    View Slide

  11. 11
    © Copyright 2015 Pivotal. All rights reserved.
    Multi-cloud
    Ÿ  Same applications running across different cloud providers:
    Private Public Hosted

    View Slide

  12. 12
    © Copyright 2015 Pivotal. All rights reserved.
    Cloud Foundry Foundation
    PLATINUM
    GOLD
    SILVER

    View Slide

  13. 13
    © Copyright 2015 Pivotal. All rights reserved.
    cf push
    Pushing your first app

    View Slide

  14. 14
    © Copyright 2015 Pivotal. All rights reserved.
    Challenge 1: Pushing your first application
    Ÿ  Choose PWS API endpoint: $ cf api https://api.run.pivotal.io
    Ÿ  Login: $ cf login
    Ÿ  Code directory: 01-simple-python-app
    Ÿ  Deploy with
    $ cf push
    Ÿ  Check it worked at http://myapp-RANDOM-WORDS.cfapps.io
    Ÿ  Turn off your app when finished with cf stop myapp (but not yet!)

    View Slide

  15. 15
    © Copyright 2015 Pivotal. All rights reserved.
    Simple Flask App Demo
    Ÿ  Simple one page “Hello World” web app
    Ÿ  Video: https://www.youtube.com/watch?v=QOfD6tnoAB8
    Ÿ  Demonstrates:
    –  Installation of requirements
    –  Scaling properties
    Ÿ  Need to Provide:
    –  App files
    –  Dependencies listed in requirements.txt file
    –  Optional manifest.yml file with configuration for deployment

    View Slide

  16. C
    F

    R
    O"
    U"
    T"
    E"
    R
    2. Set up domain
    Cloud
    Controller
    Instance
    1. Upload code
    4. Copy app into
    containerised
    instances
    3. Install Python
    &
    Dependencies
    5. Start app
    and accept
    connections
    Send request to URL
    WHAT JUST
    HAPPENED?
    Source
    Code
    Instance
    $ cf push
    Browser
    5. Load balance
    between
    instances

    View Slide

  17. 17
    © Copyright 2015 Pivotal. All rights reserved.
    What just happened?
    1.  Application code is uploaded to CF
    2.  Domain URL is set up ready for routing
    3.  Cloud controller builds application in container:
    –  Python interpreter selected
    –  Dependencies installed with pip
    4.  Container is replicated to provide instances
    5.  App starts and Router load balances requests
    Ÿ  See what’s happening using logs: $ cf logs myapp --recent

    View Slide

  18. 18
    © Copyright 2015 Pivotal. All rights reserved.
    Python on Cloud Foundry
    Ÿ  First class language (with Go, Java, Ruby, Node.js, PHP)
    Ÿ  Automatic app type detection
    –  Looks for requirements.txt or setup.py
    Ÿ  Buildpack takes care of
    –  Detecting that a Python app is being pushed
    –  Installing Python interpreter
    –  Installing packages in requirements.txt using pip
    –  Starting web app as requested (e.g. python myapp.py)

    View Slide

  19. 19
    © Copyright 2015 Pivotal. All rights reserved.
    Scaling memory and instances
    Ÿ  We can scale our application as needed in terms of memory
    and number of instances:
    $ cf scale myapp –i 5
    $ cf scale myapp –m 256M
    Ÿ  Check app in browser to see different instances being used.
    Ÿ  Scale back down with $ cf scale myapp –i 1

    View Slide

  20. 20
    © Copyright 2015 Pivotal. All rights reserved.
    How does this work?
    Ÿ  Containerised application is cached and ready to be
    deployed.
    Ÿ  Scaling number of instances replicates container and load
    balances requests across all instances.
    Ÿ  Scaling memory requires restarting app.
    Ÿ  Auto-scaling is also possible.

    View Slide

  21. 21
    © Copyright 2015 Pivotal. All rights reserved.
    Buildpacks
    Pushing data science apps

    View Slide

  22. 22
    © Copyright 2015 Pivotal. All rights reserved.
    What are buildpacks?
    Ÿ  Idea and format from Heroku
    Ÿ  Responsible for doing whatever is necessary to get your
    app running.
    Ÿ  Buildpacks take care of
    –  Detecting which type of application is being pushed
    –  Installing the appropriate run-time
    –  Installing required dependencies or other artifacts
    –  Starting the application as requested

    View Slide

  23. 23
    © Copyright 2015 Pivotal. All rights reserved.
    Containers vs Buildpacks
    runtime layer
    OS image
    application layer
    Container (e.g. Docker)
    system brings fixed
    host OS Kernel
    * Devs may bring a custom
    buildpack
    runtime layer*
    OS image
    application layer
    Buildpack
    App container
    System Provides
    Dev Provides
    system brings fixed
    host OS Kernel

    View Slide

  24. 24
    © Copyright 2015 Pivotal. All rights reserved.
    Community Buildpacks
    https://github.com/cloudfoundry-community/
    cf-docs-contrib/wiki/Buildpacks
    Ÿ  Lots of languages: 

    Clojure, Haskell, .NET, Erlang, Elixir, etc.
    Ÿ  RShiny app buildpack: 

    https://github.com/alexkago/cf-buildpack-r
    Ÿ  Can also use some Heroku buildpacks without modification.

    View Slide

  25. 25
    © Copyright 2015 Pivotal. All rights reserved.
    Using conda for package management
    Ÿ  http://conda.pydata.org
    Ÿ  Benefits:
    –  Uses precompiled binary packages
    –  No fiddling with Fortran or C compilers and library paths
    –  Known good combinations of main package versions
    –  Really simple environment management (better than virtualenv)
    –  Easy to run Python 2 and 3 side-by-side
    Go try it out if you haven’t already!

    View Slide

  26. 26
    © Copyright 2015 Pivotal. All rights reserved.
    Challenge 2: Pushing a PyData app
    Ÿ  Code directory: 02-pydata-spyre-app
    Ÿ  Spyre – Adam Hajari https://github.com/adamhajari/spyre
    Ÿ  This app uses the Pydata buildpack to install Matplotlib,
    NumPy and more.
    Ÿ  Spyre provides a simple way to build interactive web based
    visualisations similar to Rshiny.

    View Slide

  27. 27
    © Copyright 2015 Pivotal. All rights reserved.
    Next Steps
    See https://github.com/ihuston/python-cf-examples for
    continuation of tutorial:
    Ÿ  How to bind data sources like Redis & PostgreSQL
    Ÿ  Build a simple machine learning API system
    Full tutorial as a video:
    https://www.youtube.com/watch?v=lp2c05yDJko

    View Slide

  28. 28
    © Copyright 2015 Pivotal. All rights reserved.
    Resources
    Ÿ  Docs: http://docs.cloudfoundry.org
    Ÿ  CF Summit videos: http://cfsummit.com
    Ÿ  Join the CF community: http://cloudfoundry.org
    Ÿ  CF meetups: http://cloud-foundry.meetup.com

    View Slide

  29. 29
    © Copyright 2015 Pivotal. All rights reserved.
    @ianhuston

    View Slide