Running Django Applications on GAE - Flexible Environment (Previously Managed VMs) by Kenneth Kinyanjui & James Mwai

Google App Engine was made Generally Available in 2011 and supported Python, Go and Java runtimes. From that time , we have seen many popular Python web applications deployed on Google App Engine’s Standard Environment. Even with the awesomeness of App Engine, Pythonistas felt limited by what the standard environment offered.

App Engine Flexible Environment (Previously called Managed VM’s) solved the pain points that developers encountered: It allows developers to customize their Python runtime and even the underlying operating system using Dockerfiles. App Engine automatically scales your web app and offers automatic load balancing.

This talk will explore how to implement microservices, authorization, SQL and noSQL databases, traffic splitting, logging, full text search, versioning, security scanning, memcache, and content delivery networks using App Engine Flexible Environment. This will be shown in practice through the deployment of a Python/Django Application on the App Engine Flexible Environment.

Transcript

1. Deploying Django on Google App Engine Flexible Environment #PyconZA Kenneth Kinyanjui and James Mwai
2. @kenju254
3. @jmwai
4. Are you ready?
5. Overview Introduction to Google App engine Setting up Google App Engine App Engine Python runtime Setting up Cloud SQL Setting up files storage on Cloud Storage Caching data with Cloud Memcache Running background tasks with cloud Pub/Sub Stack Driver Debugger
6. Lets go back in Time
7. Evolution
8. Before
9. Engineers would just code Ops guys would deploy on servers Servers would be onsite Architectures were much simpler Take a while to ship a deployment Very tough and expensive to maintain a data centre
10. Now
11. Servers are on the cloud Ops guys and developers are working together Faster rollouts and deployments More complicated deployment architectures
12. Do we have the Money and time?
13. What? Google's Platform as a Service Runs on Google's Infrastructure Has support for multiple run times No Ops
14. What is App Engine?
16. Sweet but the catch ?
17. Standard Environment Specified runtimes ; Java, Python, Go , PHP Cannot right to the file system Preselected libraries supported Work around Google App Engine Standard Environment.
18. But Google Listened
19. Flexible Environment Google Compute Engine instance Leverage on Google App Engine No Ops Support for more runtimes ; Ruby, NodeJS Microservices on the fly with App Engine Services
20. Its Google Platform as a Service No Ops Microservices ready Pay by the hour as well
21. Cloud SQL
22. Mysql as a Service on Google Cloud Platfrom. Pay by hour. Easily connect to your services. Easily scale your instance . Easily replicate and back up your database.
23. Create a Project
24. Create a Mysql instance
25. Configuring your instance
26. DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': '<your-database-name>', 'USER': '<your-database-user>', 'PASSWORD': '<your-database-password>', 'HOST': '<your-database-host>', 'PORT': '3306', } } production settings.py
27. django-admin startproject pyconza python manage.py startapp django_gae
28. runtime: python vm: true entrypoint: gunicorn -b :$PORT mysite.wsgi runtime_config: python_version: 3 app.yaml
29. service: pyconza-demo runtime: python vm: true entrypoint: gunicorn -b :$PORT config.wsgi runtime_config: python_version: 3 env_variables: DJANGO_SETTINGS_MODULE: 'config.settings.production' CLOUD_SQL_DATABASE_USER: 'pyconza' CLOUD_SQL_DATABASE_PASSWORD: '@pyconz@' CLOUD_SQL_DATABASE_HOST: '173.194.243.95' CLOUD_SQL_DATABASE_NAME: 'pycondb' app.yaml
30. import os DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': os.environ['CLOUD_SQL_DATABASE_NAME'], 'HOST': os.environ['CLOUD_SQL_DATABASE_HOST'], 'PORT': 3306, 'USER': os.environ['CLOUD_SQL_DATABASE_USER'], 'PASSWORD': os.environ['CLOUD_SQL_DATABASE_PASSWORD'] }}
31. Cloud Storage
32. Storing Data Google cloud has a lot of storage services that you can use to suit your various storage requirements.
33. Storing Data
34. Cloud Storage Google Cloud Storage is a powerful and easy to use storage service
35. Cloud Storage Google Cloud Storage is a powerful and easy to use storage service Use it to store all your media(Videos, photos, audio) and static files(js,css).
36. Cloud Storage Google Cloud Storage is a powerful and easy to use storage service Use it to store all your media(Videos, photos, audio) and static files(js,css). You only pay for what you use.
37. Cloud Storage Google Cloud Storage is a powerful and easy to use storage service Use it to store all your media(Videos, photos, audio) and static files(js,css). You only pay for what you use. Standard storage fees about $0.026 (GB/Month)
38. Setting up Cloud Storage for your app Creating a bucket
39. Setting up Cloud Storage for your app Using the gsutil tool gsutil is a Python application that lets you access Cloud Storage from the command line. To access gsutil you need to install the Google Cloud SDK. curl https://sdk.cloud.google.com | bash
40. Setting up Cloud Storage for your app Using the gsutil tool You can use gsutil to do a wide range of bucket and object management tasks, including: Creating and deleting buckets.
41. Setting up Cloud Storage for your app Using the gsutil tool You can use gsutil to do a wide range of bucket and object management tasks, including: Creating and deleting buckets. Uploading, downloading, and deleting objects.
42. Setting up Cloud Storage for your app Using the gsutil tool You can use gsutil to do a wide range of bucket and object management tasks, including: Creating and deleting buckets. Uploading, downloading, and deleting objects. Listing buckets and objects.
43. Setting up Cloud Storage for your app Using the gsutil tool You can use gsutil to do a wide range of bucket and object management tasks, including: Creating and deleting buckets. Uploading, downloading, and deleting objects. Listing buckets and objects. Moving, copying, and renaming objects.
44. Setting up Cloud Storage for your app Using the gsutil tool You can use gsutil to do a wide range of bucket and object management tasks, including: Creating and deleting buckets. Uploading, downloading, and deleting objects. Listing buckets and objects. Moving, copying, and renaming objects. Editing object and bucket ACLs.
45. Setting up Cloud Storage for your app Using the gsutil tool gsutil help gsutil help cp gsutil mb -l gs://test-bucket gsutil ls gs://test-bucket gsutil cp file.txt gs://test-bucket gsutil mv ./dir gs://test-bucket https://cloud.google.com/storage/docs/gsutil/
46. Storing files in Cloud Storage Adding cloud storage to your python app from google.cloud import storage def upload_file(file_stream, filename, content_type): client = storage.Client(project='PROJECT_ID') bucket = client.get_bucket('bucket_id') blob = bucket.blob(filename) blob.upload_from_string( file_stream, content_type=content_type) url = blob.public_url return url
47. Storing files in Cloud Storage Using Django Storages pip install django-storages #settings.py DEFAULT_FILE_STORAGE = 'storages.backends.gs.GSBotoStorage' STATICFILES_STORAGE = 'storages.backends.gs.GSBotoStorage' GS_ACCESS_KEY_ID = os.environ['GS_ACCESS_KEY_ID'] GS_SECRET_ACCESS_KEY = os.environ['GS_SECRET_ACCESS_KEY'] GS_BUCKET_NAME = os.environ['GS_BUCKET_NAME'] STATIC_URL = "http://storage.googleapis.com/{}/static/"\ .format(GS_BUCKET_NAME)
48. Caching
49. Cloud Memcache Distributed managed data cache
50. Cloud Memcache Distributed managed data cache Memcache can be used to speed up requests in a web app.
51. Cloud Memcache Distributed managed data cache Memcache can be used to speed up requests in a web app. An app can cache query data and and subsequent requests will fetch that data from memcache and only make a query if the data is absent.
52. Cloud Memcache Distributed managed data cache Memcache can be used to speed up requests in a web app. An app can cache query data and and subsequent requests will fetch that data from memcache and only make a query if the data is absent. Data can be made to expire from the Memcache at any time.
53. Cloud Memcache There are two types of memcache on App Engine: Shared memcache - Shared by default by all app engine applications. Provides cache capacity on a best effort basis. No space or performance guarantee
54. Cloud Memcache There are two types of memcache on App Engine: Shared memcache - Shared by default by all app engine applications. Provides cache capacity on a best effort basis. No space or performance guarantee Dedicated memcache - Provides a fixed cache capacity exclusively for your application. Billed by GB-hour of cache size.
55. Cloud Memcache Limits The maximum size of a cached data value is 1 MB (10^6 bytes).
56. Cloud Memcache Limits The maximum size of a cached data value is 1 MB (10^6 bytes). A key cannot be larger than 250 bytes.
57. Cloud Memcache Limits The maximum size of a cached data value is 1 MB (10^6 bytes). A key cannot be larger than 250 bytes. The total size of the call and the total size of the data fetched must not exceed 32 megabytes.
58. Cloud Memcache Limits The maximum size of a cached data value is 1 MB (10^6 bytes). A key cannot be larger than 250 bytes. The total size of the call and the total size of the data fetched must not exceed 32 megabytes. A memcache key cannot contain a null byte.
59. Using Cloud Memcache
60. Using Cloud Memcache
61. The memcache pattern The application receives a query from the user or the application.
62. The memcache pattern The application receives a query from the user or the application. The application checks whether the data needed to satisfy that query is in memcache.
63. The memcache pattern The application receives a query from the user or the application. The application checks whether the data needed to satisfy that query is in memcache. If the data is in memcache, the application uses that data.
64. The memcache pattern The application receives a query from the user or the application. The application checks whether the data needed to satisfy that query is in memcache. If the data is in memcache, the application uses that data. If the data is not in memcache, the application queries the datastore and stores the results in memcache for future requests.
65. Cache pattern def get_data(): data = memcache.get('key') if data is not None: return data else: data = query_for_data() memcache.add('key', data, 60) return data
66. Caching values add() We use add to add values to the cache memcache.add(key="[KEY]", value="[VALUE]", time=[EXPIRATION_TIME]) memcache.add(key="weather_051016", value="cold", time=3600)
67. Retreiving values get() We use get to retrieve data from the cache memcache.get(key="[KEY]") memcache.get(key="weather_NAIROBI_280916")
68. Monitoring your cache
69. Monitoring your cache Google Cloud console can give a lot of metrics on your cached data. Such metrics include;
70. Monitoring your cache Google Cloud console can give a lot of metrics on your cached data. Such metrics include; Memcache service level: Shows if your application is using the Shared or Dedicated service level. If you are an owner of the project, you can switch between the two. Learn more about the . service levels
71. Monitoring your cache Google Cloud console can give a lot of metrics on your cached data. Such metrics include; Memcache service level: Shows if your application is using the Shared or Dedicated service level. If you are an owner of the project, you can switch between the two. Learn more about the . Hit ratio: Shows a percentage and the raw number of memcache hits and misses. service levels
72. Monitoring your cache Google Cloud console can give a lot of metrics on your cached data. Such metrics include; Memcache service level: Shows if your application is using the Shared or Dedicated service level. If you are an owner of the project, you can switch between the two. Learn more about the . Hit ratio: Shows a percentage and the raw number of memcache hits and misses. Items in the cache. service levels
73. Monitoring your cache Google Cloud console can give a lot of metrics on your cached data. Such metrics include; Memcache service level: Shows if your application is using the Shared or Dedicated service level. If you are an owner of the project, you can switch between the two. Learn more about the . Hit ratio: Shows a percentage and the raw number of memcache hits and misses. Items in the cache. Oldest item age: The age of the oldest cached item. Note that the age of an item is reset every time it is used, either read or written. service levels
74. Monitoring your cache Google Cloud console can give a lot of metrics on your cached data. Such metrics include; Memcache service level: Shows if your application is using the Shared or Dedicated service level. If you are an owner of the project, you can switch between the two. Learn more about the . Hit ratio: Shows a percentage and the raw number of memcache hits and misses. Items in the cache. Oldest item age: The age of the oldest cached item. Note that the age of an item is reset every time it is used, either read or written. Total cache size. service levels
75. Cloud Memcache Best Practises Handle memcache API failures gracefully. Memcache operations can fail for various reasons. Applications should be designed to catch failed operations without exposing these errors to end users.
76. Cloud Memcache Best Practises Handle memcache API failures gracefully. Memcache operations can fail for various reasons. Applications should be designed to catch failed operations without exposing these errors to end users. Use the batching capability of the API when possible, especially for small items. Doing so increases the performance and efficiency of your app.
77. Cloud Memcache Best Practises Handle memcache API failures gracefully. Memcache operations can fail for various reasons. Applications should be designed to catch failed operations without exposing these errors to end users. Use the batching capability of the API when possible, especially for small items. Doing so increases the performance and efficiency of your app. Distribute load across your memcache keyspace. Having a single or small set of memcache items represent a disproportionate amount of traffic will hinder your app from scaling. magnitude less than the per-GB rating.
78. Cloud Pub/Sub
79. What is Cloud Pub/Sub Asynchronous messaging that decouples senders and receivers.
80. What is Cloud Pub/Sub Asynchronous messaging that decouples senders and receivers. Uses the publisher/subscriber model for secure and highly available communication between independently written applications.
81. What is Cloud Pub/Sub Asynchronous messaging that decouples senders and receivers. Uses the publisher/subscriber model for secure and highly available communication between independently written applications. Tasks are stored as topics
82. What is Cloud Pub/Sub Asynchronous messaging that decouples senders and receivers. Uses the publisher/subscriber model for secure and highly available communication between independently written applications. Tasks are stored as topics Uses push and pull to publish and subcribe for messages from topics.
83. What is Cloud Pub/Sub Asynchronous messaging that decouples senders and receivers. Uses the publisher/subscriber model for secure and highly available communication between independently written applications. Tasks are stored as topics Uses push and pull to publish and subcribe for messages from topics. We can use in our web apps to run any number of background services.
84. What is Cloud Pub/Sub Asynchronous messaging that decouples senders and receivers. Uses the publisher/subscriber model for secure and highly available communication between independently written applications. Tasks are stored as topics Uses push and pull to publish and subcribe for messages from topics. We can use in our web apps to run any number of background services. Decouple your system into microservices and let them communicate with each other via messaging
85. Concepts Topic - A named resource to which messages are sent by publishers.
86. Concepts Topic - A named resource to which messages are sent by publishers. Subscription - A named resource representing the stream of messages from a single, specific topic, to be delivered to the subscribing application.
87. Concepts Topic - A named resource to which messages are sent by publishers. Subscription - A named resource representing the stream of messages from a single, specific topic, to be delivered to the subscribing application. Message - The combination of data and (optional) attributes that a publisher sends to a topic and is eventually delivered to subscribers.
90. Using Pub/Sub from gcloud import pubsub def create_topic(topic_name): """Create a new Pub/Sub topic.""" pubsub_client = pubsub.Client() topic = pubsub_client.topic(topic_name) topic.create() print('Topic {} created.'.format(topic.name)) Creating a topic
91. Using Pub/Sub from gcloud import pubsub def publish_message(topic_name, data): pubsub_client = pubsub.Client() topic = pubsub_client.topic(topic_name) # Data must be a bytestring data = data.encode('utf-8') message_id = topic.publish(data) print('Message {} published.'.format(message_id)) Publish messages to a topic
92. Using Pub/Sub from gcloud import pubsub def list_topics(): """ Lists all Pub/Sub topics in the current project. """ pubsub_client = pubsub.Client() topics = pubsub_client.list_topics() return topics Listing available topics
93. Using Pub/Sub from gcloud import pubsub def delete_topic(topic_name): """Deletes an existing Pub/Sub topic.""" pubsub_client = pubsub.Client() topic = pubsub_client.topic(topic_name) topic.delete() print('Topic {} deleted.'.format(topic.name)) Deleting a topic
94. Using Pub/Sub from gcloud import pubsub def create_subscription(topic_name, subscription_name): """Create a new pull subscription on the given topic.""" pubsub_client = pubsub.Client() topic = pubsub_client.topic(topic_name) subscription = topic.subscription(subscription_name) subscription.create() print('Subscription {} created on topic {}.'.format( subscription.name, topic.name)) Subscribe to a topic
95. Using Pub/Sub from gcloud import pubsub def create_subscription(topic_name, subscription_name): """Create a new pull subscription on the given topic.""" pubsub_client = pubsub.Client() topic = pubsub_client.topic(topic_name) subscription = topic.subscription(subscription_name) subscription.create() print('Subscription {} created on topic {}.'.format( subscription.name, topic.name)) Subscribe to a topic
96. Stackdriver debugger Cloud feature that lets you inspect the state of an application at any time.
97. Stackdriver debugger Cloud feature that lets you inspect the state of an application at any time. No need to use logging statements.
98. Stackdriver debugger Cloud feature that lets you inspect the state of an application at any time. No need to use logging statements. No stopping or slowing down your app
99. Stackdriver debugger Cloud feature that lets you inspect the state of an application at any time. No need to use logging statements. No stopping or slowing down your app
100. Stackdriver debugger Cloud feature that lets you inspect the state of an application at any time. No need to use logging statements. No stopping or slowing down your app pip install google-python-cloud-debugger #In your main function try: import googleclouddebugger googleclouddebugger.AttachDebugger( version='[VERSION]', module='[MODULE]' ) except ImportError: pass
101. Stackdriver debugger Cloud feature that lets you inspect the state of an application at any time. No need to use logging statements. No stopping or slowing down your app pip install google-python-cloud-debugger #In your main function try: import googleclouddebugger googleclouddebugger.AttachDebugger( version='[VERSION]', module='[MODULE]' ) except ImportError: pass
102. Thank you