Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python & Google Cloud Platform

Python & Google Cloud Platform

Сергей Пронин (CTO @ App in the Air) @ Moscow Python Conf 2017
"В рамках доклада расскажу про шестилетний опыт работы на Python с AppEngine и Google Cloud Platform.
Отличие сервисов, вариантов хранения данных, интеграция между ними; использование Google APIs.
Плюсы / минусы / подводные камни, сравнение с другими облачными платформами.
Другими словами, «Как за 6 лет вырастить стартап с нуля до трех миллионов пользователей, не думая о scaling-е»".
Видео: https://conf.python.ru/python-google-cloud-platform/

Moscow Python Meetup

October 20, 2017
Tweet

More Decks by Moscow Python Meetup

Other Decks in Programming

Transcript

  1. Google Cloud Platform

    Python
    Сергей Пронин

    App in the Air

    View full-size slide

  2. Интро
    CTO, Co-Founder

    App in the Air
    Senior Developer

    Empatika
    Python, Swift, JS, Obj-C
    GCP user from 2010

    Программная инженерия НИУ ВШЭ

    View full-size slide

  3. План
    •GCP Intro
    •Standard AppEngine Python Environment
    •Custom Environment
    •Storage
    •Services

    View full-size slide

  4. Google Cloud Platform
    • Начиналось с AppEngine Java/Python
    • Полноценная облачная платформа
    • Fully-Managed — просто заплати
    • Полноценная поддержка любых конфигураций, начиная с
    самых простых

    View full-size slide

  5. AppEngine
    • PaaS на Google-инфрасткрутуре
    • Load Balancing, Auto-Scaling, Health Checking, Instances management
    • Front-End / Back-End
    • Полная интеграция с GCP сервисами
    • Storage, Memcache, Logs, Crons, Task Queue, Email

    View full-size slide

  6. Front-End
    • Дешёвые инстансы
    • 60 сек на запрос / 600 сек на Task Queue
    • Нет background threads
    • Automatic Scaling
    • 28-instance часов бесплатно
    • Pure Python 2.7 на Standard Environment

    View full-size slide

  7. Front-End
    Type RAM CPU
    F1 128 MB 600 MHz
    F2 256 MB 1.2 GHz
    F4 512 MB 2.4 GHz
    F4_1G 1024 MB 2.4 GHz

    View full-size slide

  8. Back-End
    • ∞ запрос / 24 ч. на Task Queue
    • Background threads
    • Manual / Basic Scaling
    • 8-instance часов бесплатно
    • Pure Python 2.7 на Standard Environment

    View full-size slide

  9. Back-End
    Type RAM CPU
    B1 128 MB 600 MHz
    B2 256 MB 1.2 GHz
    B4 512 MB 2.4 GHz
    B4_1G 1024 MB 2.4 GHz
    B8 1024 MB 4.8 GHz

    View full-size slide

  10. Python Standard Env
    • Pure Python 2.7 libs
    • Runs in Sandbox, 0 instance min
    • Прямой доступ к GCP API
    • WSGI-compatible
    • webapp2, flask, django, jinja2
    • lxml, MySQLdb, PIL, ssl, pycrypto, webob, werkzeug, …

    View full-size slide

  11. class UserAvatarHandler(webapp2.RequestHandler, APIHandler):
    @user_required
    def post(self):
    avatar_file = self.request.POST.get('avatar')
    avatar = self.user.add_avatar(avatar_file)
    if not avatar:
    return self.error("bad file", code=400)
    self.success({"avatar": avatar.to_dict()})
    app = webapp2.WSGIApplication([
    ('/', WebIndexHandler),
    ('/oauth/authorize', OAuthAuthorizeHandler),
    ('/oauth/token', OAuthTokenHandler)
    #...

    ])

    View full-size slide

  12. app.yaml
    application: my-app
    version: production
    runtime: python27
    threadsafe: true
    api_version: 1
    handlers:
    - url: /_ah/mail/.+
    script: myflights.app
    login: admin
    - url: .*
    script: main.app

    libraries:
    - name: jinja2
    version: latest

    View full-size slide

  13. cron.yaml
    cron:
    - description: Update lounges
    url: /cron/lounges
    schedule: every day 00:00
    target: backend

    View full-size slide

  14. Python Flexible Env
    • Custom Python 2/3 env
    • Custom Docker supported
    • Runs on VM, 1 instance min
    • GCP API через client libraries
    • Re-use инфраструктуры Standard AppEngine
    • Ваш собственный requirement.txt

    View full-size slide

  15. app.yaml
    service: my-flex
    runtime: python
    env: flex
    entrypoint: gunicorn -c gunicorn.conf.py -b :$PORT app:app

    View full-size slide

  16. Cloud DataStore
    • Распределенное No-SQL хранилище
    • Строгие ограничения на запросы
    • Автоматическое кеширование

    View full-size slide

  17. class Airport(ndb.Model):
    name = ndb.StringProperty()
    code = ndb.StringProperty()
    city = ndb.StringProperty()
    country = ndb.StringProperty()
    offset = ndb.FloatProperty()
    phone = ndb.StringProperty()
    url = ndb.StringProperty()
    loc = ndb.GeoPtProperty()
    airport_map = ndb.StructuredProperty(AirportMap)
    terminal_maps = ndb.StructuredProperty(AirportMap, repeated=True)
    @classmethod
    def get_by_code(cls, code):
    airport = memcache.get('airport-{}'.format(code.encode('utf-8')))
    if not airport:
    airport = cls.query(cls.code == code).get()
    if airport:
    airport.set_memcache()
    return airport

    airport = Airport(name='Domodedovo', code=‘DME')
    airport.put()

    View full-size slide

  18. class Airport(ndb.Model):
    name = ndb.StringProperty()
    code = ndb.StringProperty()
    city = ndb.StringProperty()
    country = ndb.StringProperty()
    offset = ndb.FloatProperty()
    phone = ndb.StringProperty()
    url = ndb.StringProperty()
    loc = ndb.GeoPtProperty()
    airport_map = ndb.StructuredProperty(AirportMap)
    terminal_maps = ndb.StructuredProperty(AirportMap, repeated=True)
    @classmethod
    def get_by_code(cls, code):
    airport = memcache.get('airport-{}'.format(code.encode('utf-8')))
    if not airport:
    airport = cls.query(cls.code == code).get()
    if airport:
    airport.set_memcache()
    return airport

    airport = Airport(name='Domodedovo', code=‘DME')
    airport.put()

    View full-size slide

  19. class Airport(ndb.Model):
    name = ndb.StringProperty()
    code = ndb.StringProperty()
    city = ndb.StringProperty()
    country = ndb.StringProperty()
    offset = ndb.FloatProperty()
    phone = ndb.StringProperty()
    url = ndb.StringProperty()
    loc = ndb.GeoPtProperty()
    airport_map = ndb.StructuredProperty(AirportMap)
    terminal_maps = ndb.StructuredProperty(AirportMap, repeated=True)
    @classmethod
    def get_by_code(cls, code):
    airport = memcache.get('airport-{}'.format(code.encode('utf-8')))
    if not airport:
    airport = cls.query(cls.code == code).get()
    if airport:
    airport.set_memcache()
    return airport

    airport = Airport(name='Domodedovo', code='DME')
    airport.put()

    View full-size slide

  20. Cloud SQL
    • MySQL / PostgreSQL
    • Auto-Replication
    • До 64vCPUs, 416GB RAM, 10TB SSD

    View full-size slide

  21. BigQuery
    •SQL-like Data Warehouse
    •Экспорт из DataStore
    •Insert-only (no deletes / updates)
    •Идеально для аналитики

    View full-size slide

  22. – Mattias P Johansson, Spotify
    “FINALLY I CAN TELL THE WORLD THAT
    BIGQUERY IS THE BEST THING THAT
    HAS EVER HAPPENED TO ME”

    View full-size slide

  23. Logging
    class LoggingHandler(webapp2.RequestHandler):
    def get(self):
    logging.debug('This is a debug message')
    logging.info('This is an info message')
    logging.warning('This is a warning message')
    logging.error('This is an error message')
    logging.critical('This is a critical message')

    View full-size slide

  24. Services
    • Compute Engine VMs by Google
    • Cloud Storage File storage
    • Container Engine kubernetes clusters
    • Spanner New generation SQL storage
    • Pub/Sub Messaging Center
    • ML Engine Run ML Models

    View full-size slide

  25. Billing
    • AppEngine Standard все сервисы бесплатно в рамках квот
    • Pay as you go / Contract

    View full-size slide