Deploy Python apps in 5 minutes with a PaaS

How can you avoid servers and get back to coding? Platform-as-a-service (PaaS) makes deployment easy. But which PaaS do you choose and how do you get started? This talk will examine several of the leading PaaS providers and discuss their pros/cons. We'll also give examples for how to deploy the same app to each of them to see the differences.

Transcript

1. DEPLOY PYTHON APPS IN 5 MINUTES WITH A PAAS Nate Aune (@natea) PyCon Santa Clara March 15, 2013 1 Sunday, March 17, 13
2. JAZKARTA (2004-NOW) 2 Sunday, March 17, 13
3. 2010-2011 • Nothing equivalent to Heroku for Django developers • Built my own PaaS (“How hard can it be?”) • Shut down last year. Increased competition from big corps meant a race to the bottom 3 Sunday, March 17, 13
4. APPSEMBLER (2012-CURRENT) 4 Sunday, March 17, 13
5. AGENDA •Why should I care? •What is a PaaS? What are the advantages? •Which PaaS should I use? •When might I not want to use a PaaS? 5 Sunday, March 17, 13
6. WHERE ARE YOU DEPLOYING/HOSTING TODAY? •Shared hosting (i.e. Webfaction) •Running your own servers (co-located)? •Using an IaaS provider (i.e. AWS or Rackspace) •Already using a platform-as-a-service (PaaS) 6 Sunday, March 17, 13
7. MEET DAVE 7 Sunday, March 17, 13
8. 8 Sunday, March 17, 13
9. CONFIGURING A SERVER •deciding what size to get (memory, disk) •getting all the dependencies installed on it •SSHing into the machine to deploy stuff (feels dirty, but he’s under time pressure) 1  day 1  day 9 Sunday, March 17, 13
10. MAINTAINING THE SERVER 1  day 1  day 1  day 10 Sunday, March 17, 13
11. SECURITY 1  day 1  day 1  day 1  day 11 Sunday, March 17, 13
12. SCALING 1  day 1  day 1  day 1  day 1  day 12 Sunday, March 17, 13
13. OH SH*T MOMENT. 13 Sunday, March 17, 13
14. te sed 6 Sysadmin $80k DBA $90k Train new guy 8 days 14 Sunday, March 17, 13
15. WHY TOUCH SERVERS IF YOU DON’T HAVE TO? 15 Sunday, March 17, 13
16. PaaS Is  it  the  promised  land? 16 Sunday, March 17, 13
17. WHAT IS A PAAS? Platform-as-a-Service enables developers to create innovative applications without operational overhead around configuration, deployment and management. 17 Sunday, March 17, 13
18. LAYERS OF INFRASTRUCTURE Source: EngineYard “PaaS - State of the Market Survey” May 2012 - http://venturebeat.com/2012/07/05/engine-yard-paas-infographic/ 18 Sunday, March 17, 13
19. Source: EngineYard “PaaS - State of the Market Survey” May 2012 - http://venturebeat.com/2012/07/05/engine-yard-paas-infographic/ 19 Sunday, March 17, 13
20. Source: EngineYard “PaaS - State of the Market Survey” May 2012 - http://venturebeat.com/2012/07/05/engine-yard-paas-infographic/ 20 Sunday, March 17, 13
21. Source: EngineYard “PaaS - State of the Market Survey” May 2012 - http://venturebeat.com/2012/07/05/engine-yard-paas-infographic/ 21 Sunday, March 17, 13
22. Source: AppFog “Evolution of the Cloud: Toward a NoOps World” Jan 2012 http://gigaom.com/cloud/why-2013-is-the-year-of-noops-for-programmers-infographic/ 22 Sunday, March 17, 13
23. EFFICIENT, ELASTIC, SECURE • Lots of applications co-located on a few servers • Drastically reduces resources • Add/remove capacity depending on load • All secured using SELinux or LXC 23 Sunday, March 17, 13
24. LET THEM DO THE BORING STUFF • Patches and updates • Migrating applications • Backups / snapshots • Configuring everything (web servers, load balancers, modules, databases) 24 Sunday, March 17, 13
25. FOLLOW ALONG! $ git clone http://github.com/appsembler/paasbakeoff $ git branch dotcloud elasticbeanstalk gae heroku * master mezzpycon openshift stackato 25 Sunday, March 17, 13
26. 26 Sunday, March 17, 13
27. DEFINE OS DEPENDENCIES www: type: python systempackages: - openoffice.org - mysql-client-5.1 5  min 27 Sunday, March 17, 13
28. BACKUPS 5m 5m https://devcenter.heroku.com/articles/heroku-postgres-fork 28 Sunday, March 17, 13
29. SSL 5m 5m 5m 29 Sunday, March 17, 13
30. SCALING 5m 5m 5m 5m 30 Sunday, March 17, 13
31. CACHING 5m 5m 5m 5m 5m 31 Sunday, March 17, 13
32. IAAS VS. PAAS IaaS - days PaaS - minutes 32 Sunday, March 17, 13
33. WHICH PAAS? •Hosted •DIY •Hybrid 33 Sunday, March 17, 13
34. WHO ARE THE PLAYERS? • CloudFoundry (open source by VMWare) • AppFog • Stackato (ActiveState) •Heroku (now Salesforce) •Dotcloud •Gondor (Python only) •OpenShift (Redhat’s open source PaaS) •App Engine (Google) •Elastic Beanstalk (Amazon) •Azure (Microsoft) 34 Sunday, March 17, 13
35. WHO ARE THE PLAYERS? • CloudFoundry (open source by VMWare) • AppFog • Stackato (ActiveState) •Heroku (now Salesforce) •Dotcloud •Gondor (Python only) •OpenShift (Redhat’s open source PaaS) •App Engine (Google) •Elastic Beanstalk (Amazon) •Azure (Microsoft) We’ll look at these ones today. 35 Sunday, March 17, 13
36. HEROKU by Salesforce One of the first PaaS, extensive add-ons catalog 36 Sunday, March 17, 13
37. 37 Sunday, March 17, 13
38. 38 Sunday, March 17, 13
39. ANATOMY OF A DJANGO APP ON HEROKU Procfile to configure process requirements.txt to define dependencies customized settings.py for Heroku if not using S3, must configure static serve 39 Sunday, March 17, 13
40. INSTALL THE HEROKU TOOLBELT Download the Heroku Toolbelt http://toolbelt.herokuapp.com/osx/download $ heroku login Enter your Heroku credentials. Email: someone@example.com Password: ****** Could not find an existing public key. Would you like to generate one? [Yn] Generating new SSH public key. Uploading ssh public key /Users/someone/.ssh/id_rsa.pub 40 Sunday, March 17, 13
41. CREATE APP AND DB $ heroku create paasbakeoff Creating paasbakeoff... done, stack is cedar http://paasbakeoff.herokuapp.com/ | git@heroku.com:paasbakeoff.git Git remote heroku added $ heroku addons:add heroku-postgresql:dev Adding heroku-postgresql:dev on paasbakeoff... done, v3 (free) Attached as HEROKU_POSTGRESQL_GREEN_URL Database has been created and is available Use `heroku addons:docs heroku-postgresql:dev` to view documentation. $ heroku pg:info === HEROKU_POSTGRESQL_GREEN_URL Plan: Dev Status: available 41 Sunday, March 17, 13
42. SET DB ENV VARIABLES $ heroku config === paasbakeoff Config Vars HEROKU_POSTGRESQL_GREEN_URL: postgres://x:y@z.com:5432/d2b3c9ichbauv0 $ heroku pg:promote HEROKU_POSTGRESQL_GREEN Promoting HEROKU_POSTGRESQL_GREEN_URL to DATABASE_URL... done $ heroku config === paasbakeoff Config Vars DATABASE_URL: postgres://x:y@z.com:5432/d2b3c9ichbauv0 42 Sunday, March 17, 13
43. ADD TO SETTINGS.PY $ heroku config:set RACK_ENV=production Set the RACK_ENV environment variable to production 43 Sunday, March 17, 13
44. USE S3 FOR SERVING STATIC AND MEDIA FILES And Sendgrid for sending emails 44 Sunday, March 17, 13
45. PROCFILE web: gunicorn_django -b 0.0.0.0:\$PORT -w 9 -k gevent --max-requests 250 \ --preload mywebsite/settings.py Configure AWS settings $ heroku config:set AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxxx $ heroku config:set AWS_SECRET_ACCESS_KEY=yyyyyyyyyyyyyyyyyyyyyyyyy $ heroku config:set AWS_STORAGE_BUCKET_NAME=zzzzzzzzzzzzz Sync, migrate, collectstatic $ heroku run python mywebsite/manage.py createdb $ heroku run python mywebsite/manage.py collectstatic 45 Sunday, March 17, 13
46. DEPLOY WITH GIT PUSH $ git push heroku master Counting objects: 8, done. Delta compression using up to 2 threads. Compressing objects: 100% (6/6), done. Writing objects: 100% (6/6), 616 bytes, done. Total 6 (delta 3), reused 0 (delta 0) -----> Heroku receiving push -----> Python/Django app detected -----> Preparing Python interpreter (2.7.2) -----> Creating Virtualenv version 1.7.2 New python executable in .heroku/venv/bin/python2.7 ... Running virtualenv with interpreter /usr/local/bin/python2.7 -----> Activating virtualenv -----> Installing dependencies using pip version 1.1 Downloading/unpacking Django==1.4.2 (from -r requirements.txt (line 1)) ... Cleaning up... -----> Collecting static files 46 Sunday, March 17, 13
47. ANATOMY OF A DJANGO APP ON HEROKU Procfile to configure process requirements.txt to define dependencies customized settings.py for Heroku 47 Sunday, March 17, 13
48. MORE HEROKU RESOURCES • Getting Started with Django on Heroku https://devcenter.heroku.com/articles/django • Heroku Hackers Guide ($10 eBook) http://theherokuhackersguide.com • Developers Guide to Running Django Applications on Heroku http://kencochrane.net/blog/2011/11/developers-guide-for-running-django-apps-on-heroku/ • django-skel http://django-skel.readthedocs.org 48 Sunday, March 17, 13
49. DOTCLOUD polyglot from the start, very flexible, most Python centric 49 Sunday, March 17, 13
50. 50 Sunday, March 17, 13
51. 51 Sunday, March 17, 13
52. ANATOMY OF A DJANGO APP ON DOTCLOUD dotcloud.yml to store config info nginx.conf to config URL rewriting wsgi.py to serve using uWSGI createdb.py to create the database mkadmin.py to make the admin user postinstall to run syncdb, collectstatic customized settings.py for Dotcloud 52 Sunday, March 17, 13
53. INSTALL THE DOTCLOUD CLIENT $ sudo pip install dotcloud $ dotcloud setup dotCloud username or email: natea@jazkarta.com Password: ==> dotCloud authentication is complete! You are recommended to run `dotcloud check` now. 53 Sunday, March 17, 13
54. CREATE AND PUSH AN APP $ dotcloud create mezz ==> Creating a sandbox application named "mezz" ==> Application "mezz" created. Connect the current directory to "mezz"? [Y/n]: ==> Connecting with the application "mezz" ==> Connected with default push options: --rsync $ dotcloud push ==> Pushing code with rsync from "./" to application mezz building file list ... done 14:17:51: [www.0] Migrating stateful data located in ~/data 14:18:05: [www.0] Launching... 14:18:07: [www.0] Re-routing traffic to the new build... 14:18:08: [www.0] Successfully installed build revision rsync instance #0 54 Sunday, March 17, 13
55. ANATOMY OF A DJANGO APP ON DOTCLOUD dotcloud.yml to store config info nginx.conf to config URL rewriting wsgi.py to serve using uWSGI createdb.py to create the database mkadmin.py to make the admin user postinstall to run syncdb, collectstatic customized settings.py for Dotcloud 55 Sunday, March 17, 13
56. DOTCLOUD.YML 56 Sunday, March 17, 13
57. DATABASE IN SETTINGS.PY 57 Sunday, March 17, 13
58. STATIC_ROOT AND MEDIA_ROOT IN 58 Sunday, March 17, 13
59. NGINX.CONF 59 Sunday, March 17, 13
60. POSTINSTALL 60 Sunday, March 17, 13
61. CREATEDB.PY AND MKADMIN.PY mkadmin.py https://github.com/dotcloud/django-on-dotcloud/blob/master/createdb.py createdb.py 61 Sunday, March 17, 13
62. MORE RESOURCES • Dotcloud Python docs http://docs.dotcloud.com/0.9/services/python/ • Dotcloud Django docs http://docs.dotcloud.com/0.9/tutorials/python/django/ • django-on-dotcloud https://github.com/dotcloud/django-on-dotcloud/ • python-on-dotcloud https://github.com/kencochrane/python-on-dotcloud 62 Sunday, March 17, 13
63. STACKATO by ActiveState Python 3, Run anywhere, New Relic integration http://appsembler.com/blog/django-deployment-using-stackato/ 63 Sunday, March 17, 13
64. 64 Sunday, March 17, 13
65. 65 Sunday, March 17, 13
66. ANATOMY OF DJANGO APP ON STACKATO stackato.yml to define services customized settings file wsgi.py to serve using uWSGI 66 Sunday, March 17, 13
67. STACKATO 67 Sunday, March 17, 13
68. 68 Sunday, March 17, 13
69. Secure using Linux Containers (LXC) 69 Sunday, March 17, 13
70. NEW RELIC 70 Sunday, March 17, 13
71. Consistent deployment at all stages of the lifecycle 71 Sunday, March 17, 13
72. STACKATO FOR DJANGO 1.Download the Stackato client http://www.activestate.com/stackato/download_client 2.Create a wsgi.py file 3.Create a requirements.txt file (if you don’t already have one) 4.Edit the DATABASES and MEDIA_ROOT settings in settings.py file 5.Create a stackato.yml file to persist the 72 Sunday, March 17, 13
73. WSGI.PY FILE 73 Sunday, March 17, 13
74. REQUIREMENTS.TXT Django==1.4.2 psycopg2==2.4.5 -e git+git@github.com:yourname/django-awesome.git#egg=django-awesome -r requirements/project.txt Or you can just reference another requirements file in your repo 74 Sunday, March 17, 13
75. DATABASE OVERRIDES 75 Sunday, March 17, 13
76. MEDIA_ROOT 76 Sunday, March 17, 13
77. TARGET AND LOGIN $ stackato target api.appsembler.net $ stackato login --email user@domain.com Attempting to login to [https://api.appsembler.net] Password: ******** Successfully logged into [https://api.appsembler.net] 77 Sunday, March 17, 13
78. INITIAL PUSH $ stackato push Would you like to deploy from the current directory ? [Yn]: Would you like to use 'paasbakeoff' as application name ? [Yn]: Detected a Python Application, is this correct ? [Yn]: Framework: python Runtime: <framework-specific default> Application Deployed URL [paasbakeoff.appsembler.net]: Application Url: paasbakeoff.appsembler.net Enter Memory Reservation [128M]: 78 Sunday, March 17, 13
79. BIND SERVICES What kind of service ? 1. filesystem 2. memcached 3. mongodb 4. mysql 5. postgresql 6. rabbitmq 7. redis Choose: 5 Specify the name of the service [postgresql-cf691]: Creating Service: OK Binding Service: OK Create another ? [yN]: Would you like to save this configuration? [yN]: y Uploading Application [paasbakeoff]: 79 Sunday, March 17, 13
80. AUTO-GENERATED STACKATO.YML FILE name: paasbakeoff instances: 1 framework: type: python mem: 128 requirements: pypm: - pillow - psycopg2 Let’s add some other requirements to be installed requirements: pypm: - pillow pip: - psycopg2 -OR- 80 Sunday, March 17, 13
81. RUNNING SYNCDB $ stackato update -n $ stackato start $ stackato logs $ stackato run python mywebsite/manage.py syncdb --noinput Creating tables ... ... Creating table django_comments Creating table django_comment_flags Creating default account (username: admin / password: default) ... Creating default Site 127.0.0.1:8000 ... 81 Sunday, March 17, 13
82. HANDLING STATIC ASSETS $ stackato run python mywebsite/manage.py collectstatic --noinput ... Copying '/app/python/lib/python2.7/site-packages/django/contrib/admin/static/admin/css/rtl.css' Copying '/app/python/lib/python2.7/site-packages/django/contrib/admin/static/admin/css/ie.css' Copying '/app/python/lib/python2.7/site-packages/django/contrib/admin/static/admin/css/forms.css' framework: type: python home-dir: app processes: web: $STACKATO_UWSGI --static-map /static=$HOME/mywebsite/static Make sure they were copied to the right place 82 Sunday, March 17, 13
83. RUN MANAGEMENT COMMANDS hooks: post-staging: - python mywebsite/manage.py syncdb --noinput - python mywebsite/manage.py collectstatic --noinput - python mywebsite/manage.py migrate --noinput requirements: pypm: - pillow - psycopg2 - south Make sure you add South to the requirements 83 Sunday, March 17, 13
84. PERSISTED FILESYSTEM FOR FILE UPLOADS services: postgresql-cf691: postgresql filesystem-paasbakeoff: filesystem MEDIA_ROOT = os.environ['STACKATO_FILESYSTEM'] Remember to set the MEDIA_ROOT in settings.py: 84 Sunday, March 17, 13
85. STACKATO.YML FILE 85 Sunday, March 17, 13
86. DIRECTORY LAYOUT 86 Sunday, March 17, 13
87. OPENSHIFT by Redhat Open source, Auto-scaling, Jenkins builds http://appsembler.com/blog/django-deployment-using-openshift/ 87 Sunday, March 17, 13
88. 88 Sunday, March 17, 13
89. setup.py instead of requirements.txt application inside of wsgi dir .htaccess to serve up static files /data/ dir to store uploaded media files Action hooks for running commands during build, deploy, post-deploy, etc. ANATOMY OF AN OPENSHIFT REPO 89 Sunday, March 17, 13
90. INSTALL THE RHC CLIENT $ sudo gem install rhc $ rhc setup ... $ rhc domain status 7 tests, 12 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications 100% passed $ ssh-add ~/.ssh/id_rsa $ ssh-agent You may need to add the SSH key and start the SSH agent 90 Sunday, March 17, 13
91. QUICK ‘N DIRTY INSTRUCTIONS $ rhc app create -a mezzanineopenshift -t python 2.6 $ rhc cartridge add -c mysql-5.1 -a mezzanineopenshift $ cd mezzanineopenshift $ git remote add paasbakeoff git://github.com/appsembler/paasbakeoff.git $ git fetch paasbakeoff $ git merge paasbakeoff/openshift 91 Sunday, March 17, 13
92. setup.py instead of requirements.txt application inside of wsgi dir .htaccess to serve up static files /data/ dir to store uploaded media files Action hooks for running commands during build, deploy, post-deploy, etc. ANATOMY OF AN OPENSHIFT REPO 92 Sunday, March 17, 13
93. /WSGI/APPLICATION 93 Sunday, March 17, 13
94. SETUP.PY 94 Sunday, March 17, 13
95. 95 Sunday, March 17, 13
96. STATIC_ROOT AND MEDIA_ROOT 96 Sunday, March 17, 13
97. /WSGI/STATIC/.HTACCESS 97 Sunday, March 17, 13
98. .OPENSHIFT/ ACTION_HOOKS/DEPLOY 98 Sunday, March 17, 13
99. .OPENSHIFT/ ACTION_HOOKS/BUILD 99 Sunday, March 17, 13
100. CREATE AND BIND THE DATABASE $ rhc cartridge add -c mysql-5.1 -a mezz Password: ****** Adding 'mysql-5.1' to application 'mezz' Success mysql-5.1 ========= Properties ========== $ rhc cartridge add -c postgresql-8.4 -a mezz Similar for PostgreSQL 100 Sunday, March 17, 13
101. GIT PUSH TO DEPLOY $ git push Counting objects: 5, done. Delta compression using up to 2 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 498 bytes, done. Total 3 (delta 1), reused 0 (delta 0) remote: restart_on_add=false remote: Waiting for stop to finish remote: Done remote: restart_on_add=false remote: ~/git/mezz.git ~/git/mezz.git remote: ~/git/mezz.git remote: Running .openshift/action_hooks/pre_build remote: setup.py found. Setting up virtualenv remote: New python executable in /var/lib/openshift/x/python-2.6/virtenv/bin/python remote: Installing setuptools............done. remote: Installing pip...............done. ... 101 Sunday, March 17, 13
102. MORE RESOURCES • Getting started with Django on OpenShift https://openshift.redhat.com/community/get-started/django • openshift-django-example https://bitbucket.org/mdoglio/openshift-django-sample • Rapid Python and Django app deployment https://openshift.redhat.com/community/blogs/rapid-python-and-django-app- deployment-to-the-cloud-with-a-paas 102 Sunday, March 17, 13
103. ELASTIC BEANSTALK by Amazon Web Services Leverages EC2 and S3, Auto-scaling 103 Sunday, March 17, 13
104. 104 Sunday, March 17, 13
105. .elasticbeanstalk directory to store config files wsgi.py file config file with database settings for RDS 105 Sunday, March 17, 13
106. APP ENGINE Task queues, Memcache, Blobstore Inbound/outbound email, sockets via Channel API by Google 106 Sunday, March 17, 13
107. 107 Sunday, March 17, 13
108. 108 Sunday, March 17, 13
109. ANATOMY OF GAE lib directory needed since GAE can’t pip install -r requirements.txt modified settings file: add lib dir to python path and DB settings config file which defines libraries and static files mappings 109 Sunday, March 17, 13
110. SETTINGS.PY 110 Sunday, March 17, 13
111. GAE Dotcloud OpenShift 111 Sunday, March 17, 13
112. stackato.yml Stackato app.yaml App Engine 112 Sunday, March 17, 13
113. postinstall Stackato OpenShift .openshift/action_hooks/deploy 113 Sunday, March 17, 13
114. Stackato OpenShift Dotcloud Heroku Python 2.7, 3.2 stackato runtimes 2.6 (2.7) 2.6.5, 2.7.2, 3.1.2, 3.2.2 2.7.2 PostgreSQL 9.1 8.4 9.0 9.1.6 MySQL 5.5 5.1 5.1 (Yes, via RDS) Persisted FS Yes Yes Yes (Yes, via S3) Redis Yes, 2.4 No Yes, 2.4.11 (Yes, via addon) MongoDB Yes, 2.0 Yes, 2.2 Yes, 2.2.1 (Yes, via addon) Memcached Yes, 1.4 No Yes (Yes, via addon) RabbitMQ Yes, 2.4 No Yes, 2.8.5 (Yes, via addon) Solr No No Yes, 3.4.0 (Yes, via Websolr) Cron Yes Yes Yes Yes Extensible Yes, apt-get install Yes, DIY cartridge Yes, custom service Yes, buildpacks WebSockets Yes Yes Yes Yes, via Pusher add-on 114 Sunday, March 17, 13
115. Google App Engine Elastic Beanstalk Python 2.7 2.6 PostgreSQL No No MySQL 5.5 via Cloud SQL Yes, via RDS Persisted FS Yes, via Blob storage Yes, via S3 Redis No No MongoDB No No Memcached Yes Yes, ElasticCache RabbitMQ No (SQS) Solr No (CloudSearch) Cron Yes, via Task Queues No Extensible No Yes, via AMI WebSockets Yes, via Channel API No 115 Sunday, March 17, 13
116. WHY NOT PAAS? • Already invested in your own infrastructure. • Need to run on servers outside U.S. • Special requirements not met by PaaS services 116 Sunday, March 17, 13
117. OPENSHIFT PRICING 117 Sunday, March 17, 13
118. DOTCLOUD PRICING 118 Sunday, March 17, 13
119. HEROKU PRICING 119 Sunday, March 17, 13
120. GOOGLE CLOUD SQL 120 Sunday, March 17, 13
121. THANKS! Questions? 121 Sunday, March 17, 13
122. MORE INFO • Wrap-up from PaaS bake-off http://appsembler.com/blog/wrap-up-from-paas-bake-off/ • Django deployment using PaaS http://appsembler.com/blog/django-deployment-using-paas/ • django-deployer https://github.com/natea/django-deployer • paasbakeoff - code examples https://github.com/appsembler/paasbakeoff/ 122 Sunday, March 17, 13