Python & Google Cloud Platform

Сергей Пронин (CTO @ App in the Air) @ Moscow Python Conf 2017
"В рамках доклада расскажу про шестилетний опыт работы на Python с AppEngine и Google Cloud Platform.
Отличие сервисов, вариантов хранения данных, интеграция между ними; использование Google APIs.
Плюсы / минусы / подводные камни, сравнение с другими облачными платформами.
Другими словами, «Как за 6 лет вырастить стартап с нуля до трех миллионов пользователей, не думая о scaling-е»".
Видео: https://conf.python.ru/python-google-cloud-platform/

Transcript

1. Google Cloud Platform

   Python
   Сергей Пронин

   App in the Air
   

   View Slide

2. Интро
   CTO, Co-Founder

   App in the Air
   Senior Developer

   Empatika
   Python, Swift, JS, Obj-C
   GCP user from 2010

   Программная инженерия НИУ ВШЭ
   

   View Slide

3. План
   •GCP Intro
   •Standard AppEngine Python Environment
   •Custom Environment
   •Storage
   •Services
   

   View Slide

4. Google Cloud Platform
   • Начиналось с AppEngine Java/Python
   • Полноценная облачная платформа
   • Fully-Managed — просто заплати
   • Полноценная поддержка любых конфигураций, начиная с
   самых простых
   

   View Slide

5. AppEngine
   • PaaS на Google-инфрасткрутуре
   • Load Balancing, Auto-Scaling, Health Checking, Instances management
   • Front-End / Back-End
   • Полная интеграция с GCP сервисами
   • Storage, Memcache, Logs, Crons, Task Queue, Email
   

   View Slide

6. Front-End
   • Дешёвые инстансы
   • 60 сек на запрос / 600 сек на Task Queue
   • Нет background threads
   • Automatic Scaling
   • 28-instance часов бесплатно
   • Pure Python 2.7 на Standard Environment
   

   View Slide

7. Front-End
   Type RAM CPU
   F1 128 MB 600 MHz
   F2 256 MB 1.2 GHz
   F4 512 MB 2.4 GHz
   F4_1G 1024 MB 2.4 GHz
   

   View Slide

8. Back-End
   • ∞ запрос / 24 ч. на Task Queue
   • Background threads
   • Manual / Basic Scaling
   • 8-instance часов бесплатно
   • Pure Python 2.7 на Standard Environment
   

   View Slide

9. Back-End
   Type RAM CPU
   B1 128 MB 600 MHz
   B2 256 MB 1.2 GHz
   B4 512 MB 2.4 GHz
   B4_1G 1024 MB 2.4 GHz
   B8 1024 MB 4.8 GHz
   

   View Slide

10. Python Standard Env
    • Pure Python 2.7 libs
    • Runs in Sandbox, 0 instance min
    • Прямой доступ к GCP API
    • WSGI-compatible
    • webapp2, flask, django, jinja2
    • lxml, MySQLdb, PIL, ssl, pycrypto, webob, werkzeug, …
    

    View Slide

11. class UserAvatarHandler(webapp2.RequestHandler, APIHandler):
    @user_required
    def post(self):
    avatar_file = self.request.POST.get('avatar')
    avatar = self.user.add_avatar(avatar_file)
    if not avatar:
    return self.error("bad file", code=400)
    self.success({"avatar": avatar.to_dict()})
    app = webapp2.WSGIApplication([
    ('/', WebIndexHandler),
    ('/oauth/authorize', OAuthAuthorizeHandler),
    ('/oauth/token', OAuthTokenHandler)
    #...

    ])
    

    View Slide

12. app.yaml
    application: my-app
    version: production
    runtime: python27
    threadsafe: true
    api_version: 1
    handlers:
    - url: /_ah/mail/.+
    script: myflights.app
    login: admin
    - url: .*
    script: main.app
    

    libraries:
    - name: jinja2
    version: latest
    

    View Slide

13. cron.yaml
    cron:
    - description: Update lounges
    url: /cron/lounges
    schedule: every day 00:00
    target: backend
    

    View Slide

14. Python Flexible Env
    • Custom Python 2/3 env
    • Custom Docker supported
    • Runs on VM, 1 instance min
    • GCP API через client libraries
    • Re-use инфраструктуры Standard AppEngine
    • Ваш собственный requirement.txt
    

    View Slide

15. app.yaml
    service: my-flex
    runtime: python
    env: flex
    entrypoint: gunicorn -c gunicorn.conf.py -b :$PORT app:app
    

    View Slide

16. Storage
    

    View Slide

17. Cloud DataStore
    • Распределенное No-SQL хранилище
    • Строгие ограничения на запросы
    • Автоматическое кеширование
    

    View Slide

18. class Airport(ndb.Model):
    name = ndb.StringProperty()
    code = ndb.StringProperty()
    city = ndb.StringProperty()
    country = ndb.StringProperty()
    offset = ndb.FloatProperty()
    phone = ndb.StringProperty()
    url = ndb.StringProperty()
    loc = ndb.GeoPtProperty()
    airport_map = ndb.StructuredProperty(AirportMap)
    terminal_maps = ndb.StructuredProperty(AirportMap, repeated=True)
    @classmethod
    def get_by_code(cls, code):
    airport = memcache.get('airport-{}'.format(code.encode('utf-8')))
    if not airport:
    airport = cls.query(cls.code == code).get()
    if airport:
    airport.set_memcache()
    return airport

    airport = Airport(name='Domodedovo', code=‘DME')
    airport.put()
    

    View Slide

19. class Airport(ndb.Model):
    name = ndb.StringProperty()
    code = ndb.StringProperty()
    city = ndb.StringProperty()
    country = ndb.StringProperty()
    offset = ndb.FloatProperty()
    phone = ndb.StringProperty()
    url = ndb.StringProperty()
    loc = ndb.GeoPtProperty()
    airport_map = ndb.StructuredProperty(AirportMap)
    terminal_maps = ndb.StructuredProperty(AirportMap, repeated=True)
    @classmethod
    def get_by_code(cls, code):
    airport = memcache.get('airport-{}'.format(code.encode('utf-8')))
    if not airport:
    airport = cls.query(cls.code == code).get()
    if airport:
    airport.set_memcache()
    return airport

    airport = Airport(name='Domodedovo', code=‘DME')
    airport.put()
    

    View Slide

20. class Airport(ndb.Model):
    name = ndb.StringProperty()
    code = ndb.StringProperty()
    city = ndb.StringProperty()
    country = ndb.StringProperty()
    offset = ndb.FloatProperty()
    phone = ndb.StringProperty()
    url = ndb.StringProperty()
    loc = ndb.GeoPtProperty()
    airport_map = ndb.StructuredProperty(AirportMap)
    terminal_maps = ndb.StructuredProperty(AirportMap, repeated=True)
    @classmethod
    def get_by_code(cls, code):
    airport = memcache.get('airport-{}'.format(code.encode('utf-8')))
    if not airport:
    airport = cls.query(cls.code == code).get()
    if airport:
    airport.set_memcache()
    return airport

    airport = Airport(name='Domodedovo', code='DME')
    airport.put()
    

    View Slide

21. Pokemon Go
    

    View Slide

22. Cloud SQL
    • MySQL / PostgreSQL
    • Auto-Replication
    • До 64vCPUs, 416GB RAM, 10TB SSD
    

    View Slide

23. BigQuery
    •SQL-like Data Warehouse
    •Экспорт из DataStore
    •Insert-only (no deletes / updates)
    •Идеально для аналитики
    

    View Slide

24. – Mattias P Johansson, Spotify
    “FINALLY I CAN TELL THE WORLD THAT
    BIGQUERY IS THE BEST THING THAT
    HAS EVER HAPPENED TO ME”
    

    View Slide

25. Logging
    class LoggingHandler(webapp2.RequestHandler):
    def get(self):
    logging.debug('This is a debug message')
    logging.info('This is an info message')
    logging.warning('This is a warning message')
    logging.error('This is an error message')
    logging.critical('This is a critical message')
    

    View Slide

26. Services
    • Compute Engine VMs by Google
    • Cloud Storage File storage
    • Container Engine kubernetes clusters
    • Spanner New generation SQL storage
    • Pub/Sub Messaging Center
    • ML Engine Run ML Models
    

    View Slide

27. Billing
    • AppEngine Standard все сервисы бесплатно в рамках квот
    • Pay as you go / Contract
    

    View Slide

28. View Slide

29. python@appintheair.mobi
    Q & A
    

    View Slide