Slide 1

Slide 1 text

Building webapps for the Cloud with Python and Google App Engine Juan Gomez PythonKC Co-founder October 29th, 2011

Slide 2

Slide 2 text

Agenda 1. Intro to Google App Engine (GAE for short) • Brief intro to Python • Structure of a Python webapp • Setting up your dev environment with GAE • App Engine Architecture • The App Engine datastore and GQL 2. Brief intro to Django • Using templates with GAE 3. Quick demo of a sample webapp 4. Beyond the basics • Scalability & Security • Quotas • Using the Google Data Services API 5. Summary • Where to go from here? • References 2

Slide 3

Slide 3 text

Why Google App Engine? Challenges building web apps 3

Slide 4

Slide 4 text

Run your web applications on Google’s infrastructure

Slide 5

Slide 5 text

Easy to start. Easy to scale.

Slide 6

Slide 6 text

The Components

Slide 7

Slide 7 text

7 1. Scalable Serving Infrastructure 2. Python Runtime 3. Software Development Kit 4. Web based Admin Console 5. Datastore

Slide 8

Slide 8 text

4. Web-based Admin Console 8

Slide 9

Slide 9 text

Click to Deploy 9

Slide 10

Slide 10 text

Memcache Send E-Mail Make HTTP Requests Authenticate with Google Accounts Image Manipulation

Slide 11

Slide 11 text

Does one thing well: running web apps

Slide 12

Slide 12 text

12

Slide 13

Slide 13 text

13

Slide 14

Slide 14 text

A Python Code Sample x = 34 - 23 # A comment. y = “Hello” # Another one. z = 3.45 if z is not 3.46 or y is “Hello”: x = x + 1 y = y + “ World” # String concat. print x print y 14

Slide 15

Slide 15 text

Enough to Understand the Code • First assignment to a variable creates it • Assignment is = and comparison is == (or is) • For numbers + - * / % are as expected • Special use: • + for string concatenation • % for string formatting (as in C’s printf) • Logical operators are words (and, or, not) not symbols (&&, ||, !). • The basic printing command is print 15

Slide 16

Slide 16 text

Comments • Start comments with #, rest of line is ignored • Can include a “documentation string” as the first line of a new function or class you define • Development environments, debugger, and other tools use it: it’s good style to include one def my_function(x, y): “““This is the docstring. This function does blah blah blah.””” # The code would go here... 16

Slide 17

Slide 17 text

Python and Types • Everything is an object! • “Dynamic Typing”-> Data types determined automatically. • “Strong Typing” -> Enforces them after it figures them out. x = “the answer is ” # Decides x is string. y = 23 # Decides y is integer. print x + y # Python will complain about this. 17

Slide 18

Slide 18 text

Basic Datatypes • Integers (default for numbers) •z = 5 / 2 # Answer 2, integer division • Floats •x = 3.456 • Strings • Can use “” or ‘’ to specify with “abc” == ‘abc’ • Unmatched can occur within the string: “matt’s” • Use triple double-quotes for multi-line strings or strings that contain both ‘ and “ inside of them: “““a‘b“c””” 18

Slide 19

Slide 19 text

Whitespace Whitespace is meaningful in Python: especially indentation and placement of newlines •Use a newline to end a line of code Use \ when must go to next line prematurely •No braces {} to mark blocks of code, use consistent indentation instead • First line with less indentation is outside of the block • First line with more indentation starts a nested block •Colons start of a new block in many constructs, e.g. function definitions, then clauses 19

Slide 20

Slide 20 text

Assignment •You can assign to multiple names at the same time >>> x, y = 2, 3 >>> x 2 >>> y 3 This makes it easy to swap values >>> x, y = y, x •Assignments can be chained >>> a = b = x = 2 20

Slide 21

Slide 21 text

A Python Code Sample x = 34 - 23 # A comment. y = “Hello” # Another one. z = 3.45 if z is not 3.46 or y is “Hello”: x = x + 1 y = y + “ World” # String concat. print x print y 21

Slide 22

Slide 22 text

Side by Side with Java 22 Java (C#) Python public class Employee { private String myEmployeeName; private int myTaxDeductions = 1; private String myMaritalStatus = "single"; public Employee(String EmployeName) { this(EmployeName, 1); } public Employee(String EmployeName, int taxDeductions) { this(EmployeName, taxDeductions, "single"); } public Employee(String EmployeName, int taxDeductions, String maritalStatus) { this.myEmployeeName = EmployeName; this.myTaxDeductions = taxDeductions; this.myMaritalStatus = maritalStatus; } } class Employee(): def __init__(self, employeeName , taxDeductions=1 , maritalStatus="single" ): self.employeeName = employeeName self.taxDeductions = taxDeductions self.maritalStatus = maritalStatus

Slide 23

Slide 23 text

23 Life is Short (You Need Python) - Bruce Eckel (Thinking in C++)

Slide 24

Slide 24 text

Things to read through: The Quick Python Book, 2nd Ed http://amzn.to/lXKzH5 “Learn Python The Hard Way” http://learnpythonthehardway.org 24 Python 101 – Beginning Python http://www.rexx.com/~dkuhlman/python_101/python_101.html

Slide 25

Slide 25 text

Things to refer to: The Python Standard Library by Example http://amzn.to/sx3It1 Programming Python, 4th Ed http://amzn.to/kWjaW2 25 The Official Python Tutorial http://www.python.org/doc/current/tut/tut.html The Python Quick Reference http://rgruet.free.fr/PQR2.3.html

Slide 26

Slide 26 text

26

Slide 27

Slide 27 text

Google App Engine

Slide 28

Slide 28 text

• App Engine handles HTTP(S) requests, nothing else – Think RPC: request in, processing, response out – Works well for the web and AJAX; also for other services • App configuration is dead simple – No performance tuning needed • Everything is built to scale – “infinite” number of apps, requests/sec, storage capacity – APIs are simple, stupid App Engine Does One Thing Well 28

Slide 29

Slide 29 text

And that allows it to: • Serve static files • Serve dynamic requests • Store data • Call web services • Authenticate against Google’s user database • Send e-mail, process images, use memcache 29

Slide 30

Slide 30 text

Scaling • 5 million page views a month at the free quota • Low-usage apps: many apps per physical host • High-usage apps: multiple physical hosts per app • Stateless APIs are trivial to replicate • Memcache is trivial to shard • Datastore built on top of Bigtable; designed to scale well – Abstraction on top of Bigtable – API influenced by scalability • No joins • Recommendations: denormalize schema; precompute joins 30

Slide 31

Slide 31 text

Python webapps • App Engine includes a simple web application framework called webapp. • webapp is a WSGI-compatible framework. • You can use webapp or any other WSGI framework with GAE (like web.py, CherryPy, Tornado, Django) • Basic apps need Config file + webapp CGI 31

Slide 32

Slide 32 text

Config file (No XML!) • A webapp specifies runtime configuration, including versions and URLs, in a file named app.yaml. application: myapp version: 1 runtime: python api_version: 1 handlers: - url: /admin/.* script: admin.py login: admin - url: /index\.html script: home.py - url: /(.*\.(gif|png|jpg)) static_files: static/\1 upload: static/(.*\.(gif|png|jpg)) 32

Slide 33

Slide 33 text

Structure of a Python webapp from google.appengine.ext import webapp from google.appengine.ext.webapp.util import run_wsgi_app class MainPage(webapp.RequestHandler): def get(self): self.response.headers['Content-Type'] = 'text/plain' self.response.out.write('Hello, webapp World!') application = webapp.WSGIApplication([('/', MainPage)], debug=True) def main(): run_wsgi_app(application) if __name__ == "__main__": main() 33

Slide 34

Slide 34 text

Setting up your dev environment with GAE • The dev environment is really, really nice • Download the (open source) SDK – http://code.google.com/appengine/downloads.html – Google_App_Engine_SDK_for_Python • a full simulation of the App Engine environment • dev_appserver.py myapp for a local webserver • appcfg.py update myapp to deploy to the cloud • You get a GUI on OS X and Windows. 34

Slide 35

Slide 35 text

App Engine Architecture 35 Python VM process stdlib app memcache datastore mail images urlfech stateful APIs stateless APIs R/O FS req/resp

Slide 36

Slide 36 text

The Datastore • Based on “BigTable” • Schemaless • Scales infinitely • NoSQL, with SQL type queries (GQL) – No joins (they do have “reference fields”) – No aggregate queries - not even count()! – Hierarchy affects sharding and transactions – All queries must run against an existing index • Maybe the best part of App Engine! 36

Slide 37

Slide 37 text

Hierarchical Datastore • Entities have a Kind, a Key, and Properties – Entity -> Record -> Python dict -> Python class instance – Key -> structured foreign key; includes Kind – Kind -> Table -> Python class – Property -> Column or Field; has a type • Dynamically typed: Property types are recorded per Entity • Key has either id or name – the id is auto-assigned; alternatively, the name is set by app – A key can be a path including the parent key, and so on • Paths define entity groups which limit transactions – A transaction locks the root entity (parentless ancestor key) 37

Slide 38

Slide 38 text

Creating an Entity in the Datastore from google.appengine.ext import db ... class FoursquareUser(db.Model): """Creation of an entity of the kind 'FoursquareUser'""" created = db.DateTimeProperty(auto_now_add=True) name = db.TextProperty() email = db.TextProperty() description = db.StringProperty(multiline=True) 38

Slide 39

Slide 39 text

GQL • GQL is a SQL-like language for retrieving entities or keys from the datastore. • GQL's features are different from a query language for a traditional relational database. • GQL syntax is (very) similar to that of SQL • SELECT [* | __key__] FROM [WHERE [AND ...]] [ORDER BY [ASC | DESC] [, [ASC | DESC] ...]] [LIMIT [,]] [OFFSET ] := {< | <= | > | >= | = | != } := IN := ANCESTOR IS 39

Slide 40

Slide 40 text

Querying my FoursquareUser Entity from google.appengine.ext import db ... # I can query the datastore using the Query API query = db.Query(FoursquareUser) # Or the methods inherited form db.Model model_query = FoursquareUser.all() # Or using GQL GQL_query = db.GqlQuery("SELECT * FROM FoursquareUser") # I can print the results to the HTTP response object for user in query: self.response.out.write("User name: %s" % user["name"]) self.response.out.write("User email: %s" % user["email"]) 40

Slide 41

Slide 41 text

Creating the Views • HTML embedded in code is messy and difficult to maintain. • It's better to use a templating system. – HTML is kept in a separate files with special syntax to indicate where the data from the application appears in the view. • There are many templating systems for Python: EZT, Cheetah, ClearSilver, Quixote, and Django are just a few. • You can use your template engine of choice by bundling it with your application code. • Or you can use Django out of the box! 41

Slide 42

Slide 42 text

It’s a magical pony!

Slide 43

Slide 43 text

Why use Django over webapp? • Django templating is one of the best in its class • Django has easy cookies and custom 500 errors • Django is less verbose • Django middleware is really handy • URL patterns – mapping between URL patterns (RegEx) to callback functions (views). – URLconf 43

Slide 44

Slide 44 text

Using templates with GAE from django.conf.urls.defaults import * from django.http import HttpResponse def hello(request): return HttpResponse("Hello, World!") urlpatterns = patterns('', ('^$', hello),) # You have to write even less code! 44

Slide 45

Slide 45 text

Quick demo of a sample webapp 45

Slide 46

Slide 46 text

Automatic Scaling to Application Needs • You don’t need to configure your resource needs • One CPU can handle many requests per second • Apps are hashed (really mapped) onto CPUs: – One process per app, many apps per CPU – Creating a new process is a matter of cloning a generic “model” process and then loading the application code (in fact the clones are pre-created and sit in a queue) – The process hangs around to handle more requests (reuse) – Eventually old processes are killed (recycle) • Busy apps (many QPS) get assigned to multiple CPUs – This automatically adapts to the need • as long as CPUs are available 46

Slide 47

Slide 47 text

Security • Prevent the bad guys from breaking (into) your app • Constrain direct OS functionality – no processes, threads, dynamic library loading (use Task Queue) – no sockets (use urlfetch API) – can’t write files (use datastore) – disallow unsafe Python extensions (e.g. ctypes) • Limit resource usage – Limit 10,000 files per app, adding up to 32 MB – Hard time limit of 60 seconds per request – Daily Max of 6.50 CPU hours (CPU cycles on 1.2 GHz Intel x86) – Hard limit of 32 MB on request and response size, API call size, etc. – Quota system for number of requests, API calls, emails sent, etc 47

Slide 48

Slide 48 text

Preserving Fairness Through Quotas • Everything an app does is limited by quotas. • If you run out of quota that particular operation is blocked • Free quotas are tuned so that a well-written app (light CPU/datastore use) can survive a moderate “slashdotting” • The point of quotas is to be able to support a very large number of small apps. • Large apps need raised quotas ($$$) 48

Slide 49

Slide 49 text

Costs Quotas 49

Slide 50

Slide 50 text

Quota’s Quotas 50

Slide 51

Slide 51 text

Quotas 51

Slide 52

Slide 52 text

Using the Google Data Service API import gdata.docs.service # Create a client class which will make HTTP requests with # Google Docs server. client = gdata.docs.service.DocsService() # Authenticate using your Google Docs email address and # password. client.ClientLogin('[email protected]', 'password') # Query the server for an Atom feed containing a list of your # documents. documents_feed = client.GetDocumentListFeed() # Loop through the feed and extract each document entry. for document_entry in documents_feed.entry: # Display the title of the document on the command line. print document_entry.title.text 52

Slide 53

Slide 53 text

53 What about vendor Lock-in?

Slide 54

Slide 54 text

What about vendor Lock-in? • Use Django-nonrel – independent branch of Django that adds NoSQL database support to the ORM http://www.allbuttonspressed.com/projects/django- nonrel – No JOINs! :-( • Port existing Django projects with very few changes! • Switch from Bigtable to other NoSQL DBs – MongoDB, SimpleDB. (soon Cassandra, CouchDB) • Move freely between Django hosting providers – ep.io, gondor.io, Heroku or your own VPS 54

Slide 55

Slide 55 text

Where to go from here? •Learn Python! •Download the App Engine SDK •Build Apps...Many Apps •Learn Django! (is awesome) •Join a local Python User Group – http://www.pyowa.org or @pyowa on Twitter. – http://pythonkc.com/ or @pythonkc on Twitter. 55

Slide 56

Slide 56 text

Resources • Read the Articles on the Google App Engine site. – http://code.google.com/appengine/docs/python/gettingstarted/ – http://code.google.com/appengine/docs/python/overview.html – http://appengine-cookbook.appspot.com/ • Keep up with new releases through the Blog – http://googleappengine.blogspot.com/ • Read the source code from the SDK – http://code.google.com/p/googleappengine/ – Maybe you will find an undocumented API. 56

Slide 57

Slide 57 text

Things to read through: Programming Google App Engine. (Python & Java) http://amzn.to/ovzX4q 57

Slide 58

Slide 58 text

Before the Q & A... 58

Slide 59

Slide 59 text

59 Yes! Slides can be downloaded here: http://www.slideshare.net/_juandg/gae_icc_fall2011

Slide 60

Slide 60 text

Q & A • Anything goes 60