Diving into Flask (Head On)

Slide 1

Slide 1 text

Diving Into Flask Head On Andrii V. Mishkovskyi contact@mishkovskyi.net A. Mishkovskyi () Diving Into Flask EuroPython 2012 1 / 40

Slide 2

Slide 2 text

Warming up Section 1 Warming up A. Mishkovskyi () Diving Into Flask EuroPython 2012 2 / 40

Slide 3

Slide 3 text

Warming up Presentation theme Share our experience with Flask Explore inner workings of libraries we used Understand why things break A. Mishkovskyi () Diving Into Flask EuroPython 2012 3 / 40

Slide 4

Slide 4 text

Warming up Why Flask? Well-documented Great API Easily extendable Well-suited for web APIs A. Mishkovskyi () Diving Into Flask EuroPython 2012 4 / 40

Slide 5

Slide 5 text

Warming up Why Flask? Yes, we also considered Django, Pyramid and many more A. Mishkovskyi () Diving Into Flask EuroPython 2012 4 / 40

Slide 6

Slide 6 text

Exploring Flask Section 2 Exploring Flask A. Mishkovskyi () Diving Into Flask EuroPython 2012 5 / 40

Slide 7

Slide 7 text

Exploring Flask Starting with the simplest Where we all start from flask import Flask app = Flask(__name__) @app.route(’/’) def hello_world (): return ’Hello World!’ if __name__ == ’__main__ ’: app.run() A. Mishkovskyi () Diving Into Flask EuroPython 2012 6 / 40

Slide 8

Slide 8 text

Exploring Flask Starting with the simplest Where we all start @app.route(’/’) def hello_world (): return ’Hello World!’ A. Mishkovskyi () Diving Into Flask EuroPython 2012 6 / 40

Slide 9

Slide 9 text

Exploring Flask Starting with the simplest Where some of us end up @app.route(’/albums/’ ’/photos//’ ’’) def photo_action(album_id , photo_id , action ): ... A. Mishkovskyi () Diving Into Flask EuroPython 2012 6 / 40

Slide 10

Slide 10 text

Exploring Flask Flask views Manual dispatch @app.route(’/foo’, methods =[’GET’, ’POST ’, ’PUT’]) def foo (): if request.method == ’GET’: return get_foo () elif request.method == ’POST ’: return create_foo () else: retturn update_foo () A. Mishkovskyi () Diving Into Flask EuroPython 2012 7 / 40

Slide 11

Slide 11 text

Exploring Flask Flask views Let Flask do all the hard work @app.route(’/foo’, methods =[’GET’]) def get_foo (): ... @app.route(’/foo’, methods =[’POST ’]) def create_foo (): ... @app.route(’/foo’, methods =[’PUT’]) def update_foo (): ... A. Mishkovskyi () Diving Into Flask EuroPython 2012 8 / 40

Slide 12

Slide 12 text

Exploring Flask Flask views Class views with manual dispatch class Foo(View ): def dispatch_request (self ): if request.method == ’GET’: return self.get() elif request.method == ’POST ’: return self.create () elif request.method == ’PUT’: return self.update () app.add_url_rule( ’/foo’, view_func=Foo.as_view(’foo’)) A. Mishkovskyi () Diving Into Flask EuroPython 2012 9 / 40

Slide 13

Slide 13 text

Exploring Flask Flask views Class views with HTTP method-based dispatch class Foo(MethodView ): def get(self ): ... def post(self ): ... def put(self ): ... app.add_url_rule( ’/foo’, view_func=Foo.as_view(’foo’)) A. Mishkovskyi () Diving Into Flask EuroPython 2012 10 / 40

Slide 14

Slide 14 text

Exploring Flask Simple, yet powerful Flask.route Decorator that calls Flask.add url rule Flask.add url rule creates werkzeug.routing.Rule and adds it to werkzeug.routing.Map werkzeug.routing.Map does the URL matching magic A. Mishkovskyi () Diving Into Flask EuroPython 2012 11 / 40

Slide 15

Slide 15 text

Exploring Flask Simple, yet powerful Class views Can’t use Flask.route decorator Explicitly call Flask.add url rule as view method with creates the actual view function A. Mishkovskyi () Diving Into Flask EuroPython 2012 12 / 40

Slide 16

Slide 16 text

Exploring Flask Simple, yet powerful Class views class View(object ): @classmethod def as_view(cls , name , *class_args , ** class_kwargs ): def view (*args , ** kwargs ): self = view.view_class( *class_args , ** class_kwargs) return self. dispatch_request ( *args , ** kwargs) view.view_class = cls view.__name__ = name view.__doc__ = cls.__doc__ view.__module__ = cls.__module__ view.methods = cls.methods return view A. Mishkovskyi () Diving Into Flask EuroPython 2012 12 / 40

Slide 17

Slide 17 text

Exploring Flask Simple, yet powerful URL matching and decomposition Rule creates regexp and collects proper converters Map holds all rules and builds the string for Rule to match Converters convert the path parts into Python objects A. Mishkovskyi () Diving Into Flask EuroPython 2012 13 / 40

Slide 18

Slide 18 text

Exploring Flask Simple, yet powerful URL matching and decomposition >>> from werkzeug.routing import Map , Rule >>> rule = Rule(’/yada/daba/’ ’’ ’/’) >>> Map([ rule ]) >>> print(rule._regex.pattern) ^\|\/ yada \/ daba \/(?P[^/]{2})\/(?P\d+)$ >>> rule._converters {’baz’: , ’bar’: } >>> rule._trace [(False , ’|’), (False , ’/yada/daba/’), (True , ’bar’), (False , ’/’), (True , ’baz’)] >>> rule._weights [(0, -4), (0, -4), (1, 100) , (1, 50)] A. Mishkovskyi () Diving Into Flask EuroPython 2012 13 / 40

Slide 19

Slide 19 text

Exploring Flask Simple, yet powerful URL matching and decomposition Rule objects are stored in Map in sorted order. class Rule(RuleFactory ): def match_compare_key (self ): return (bool(self.arguments), -len(self._weights), self._weights) # Somewhere in Map implementation self._rules.sort( key=lambda x: x. match_compare_key ()) A. Mishkovskyi () Diving Into Flask EuroPython 2012 13 / 40

Slide 20

Slide 20 text

Exploring Flask Blueprints Modular Flask More manageable No more interference with other’s work Pluggable views Turnkey functionality implementations A. Mishkovskyi () Diving Into Flask EuroPython 2012 14 / 40

Slide 21

Slide 21 text

Exploring Flask Blueprints Introducing blueprints We needed API versioning Instant win: url prefix Also splitting admin and API endpoints Ability to deﬁne per-blueprint template folder A. Mishkovskyi () Diving Into Flask EuroPython 2012 15 / 40

Slide 22

Slide 22 text

Exploring Flask Blueprints How blueprints work Basically a proxy object That tracks if it was registered before The only interesting details is URL registration A. Mishkovskyi () Diving Into Flask EuroPython 2012 16 / 40

Slide 23

Slide 23 text

Exploring Flask Blueprints How blueprints work from flask import Blueprint API = Blueprint( ’API’, __name__ , url_prefix=’/api/v1’) @API.route(’/foo’) def foo (): ... A. Mishkovskyi () Diving Into Flask EuroPython 2012 16 / 40

Slide 24

Slide 24 text

Exploring Flask Blueprints How blueprints work class Blueprint( _PackageBoundObjects ): def record(self , func ): ... self. deferred_functions .append(func) def add_url_rule (self , rule , endpoint=None , view_func=None , ** options ): ... self.record(lambda s: s.add_url_rule(rule , endpoint , view_func , ** options )) A. Mishkovskyi () Diving Into Flask EuroPython 2012 16 / 40

Slide 25

Slide 25 text

Exploring Flask Blueprints How blueprints work class Flask( _PackageBoundObject ): def register_blueprint (self , blueprint , ** options ): ... blueprint.register(self , options) class Blueprint( _PackageBoundObjects ): def register(self , app , options ): ... state = self. make_setup_state (app , options) for deferred in self. deferred_functions : deferred(state) A. Mishkovskyi () Diving Into Flask EuroPython 2012 16 / 40

Slide 26

Slide 26 text

Flask and SQLAlchemy Section 3 Flask and SQLAlchemy A. Mishkovskyi () Diving Into Flask EuroPython 2012 17 / 40

Slide 27

Slide 27 text

Flask and SQLAlchemy Overview Flask-SQLAlchemy Full of magic As in, dark magic Say, would you guess what is the purpose of this? A. Mishkovskyi () Diving Into Flask EuroPython 2012 18 / 40

Slide 28

Slide 28 text

Flask and SQLAlchemy Overview Flask-SQLAlchemy def _calling_context (app_path ): frm = sys._getframe (1) while frm.f_back is not None: name = frm.f_globals.get(’__name__ ’) if name and \ (name == app_path or name.startswith(app_path + ’.’)): funcname = frm.f_code.co_name return ’%s:%s (%s)’ % ( frm.f_code.co_filename , frm.f_lineno , funcname ) frm = frm.f_back return ’’ A. Mishkovskyi () Diving Into Flask EuroPython 2012 18 / 40

Slide 29

Slide 29 text

Flask and SQLAlchemy Overview Flask-SQLAlchemy A. Mishkovskyi () Diving Into Flask EuroPython 2012 18 / 40

Slide 30

Slide 30 text

Flask and SQLAlchemy Partitioning SQLAlchemy and binds Bind is the SQLAlchemy engine or pure connection object Flask-SQLAlchemy gives the ability to specify bind per model But sometimes one model has to reference several binds A. Mishkovskyi () Diving Into Flask EuroPython 2012 19 / 40

Slide 31

Slide 31 text

Flask and SQLAlchemy Partitioning SQLAlchemy and binds class AdminUsers(db.Model ): __bind_key__ = ’admin ’ # model definition goes here A. Mishkovskyi () Diving Into Flask EuroPython 2012 19 / 40

Slide 32

Slide 32 text

Flask and SQLAlchemy Partitioning How Flask-SQLAlchemy does it def get_bind(self , mapper , clause=None ): if mapper is not None: info = getattr( mapper.mapped_table , ’info ’, {}) bind_key = info.get(’bind_key ’) if bind_key is not None: state = get_state(self.app) return state.db.get_engine( self.app , bind=bind_key) return Session.get_bind(self , mapper , clause) A. Mishkovskyi () Diving Into Flask EuroPython 2012 20 / 40

Slide 33

Slide 33 text

Flask and SQLAlchemy Partitioning How do we achieve master-slave support? db.session.using_bind(’slave ’). query (...) db.session.using_bind(’master ’). query (...) AdminUser.query_using(’admin -slave -1’).all() AdminUser.query_using(’admin -slave -2’).all() A. Mishkovskyi () Diving Into Flask EuroPython 2012 21 / 40

Slide 34

Slide 34 text

Flask and SQLAlchemy Partitioning How do we achieve master-slave support? def __init__(self , *args , ** kwargs ): _SignallingSession .__init__( self , *args , ** kwargs) self._name = None def using_bind(self , name ): self._name = name return self A. Mishkovskyi () Diving Into Flask EuroPython 2012 21 / 40

Slide 35

Slide 35 text

Flask and SQLAlchemy Partitioning How do we achieve master-slave support? def get_bind(self , mapper , clause=None ): if mapper is not None: info = getattr(mapper.mapped_table , ’info ’, {}) bind_key = self._name or \ info.get(’bind_key ’) else: bind_key = self._name if bind_key is not None: state = get_state(self.app) return state.db.get_engine( self.app , bind=bind_key) else: return Session.get_bind( self , mapper , clause) A. Mishkovskyi () Diving Into Flask EuroPython 2012 21 / 40

Slide 36

Slide 36 text

Flask and SQLAlchemy Migrations SQLAlchemy-migrate Easy to start with Decent documentation Seems abandoned Had to write a wrapper to run migrate utility A. Mishkovskyi () Diving Into Flask EuroPython 2012 22 / 40

Slide 37

Slide 37 text

Flask and SQLAlchemy Migrations Alembic 7 months ago seemed to be in alpha state Much more mature right now Great documentation, great implementation Written by Mike Bayer himself A. Mishkovskyi () Diving Into Flask EuroPython 2012 23 / 40

Slide 38

Slide 38 text

Deferring your tasks Section 4 Deferring your tasks A. Mishkovskyi () Diving Into Flask EuroPython 2012 24 / 40

Slide 39

Slide 39 text

Deferring your tasks Introducing celery Celery features Removes the hassle of using amqplib/pika Extensive set of features Confusing documentation A. Mishkovskyi () Diving Into Flask EuroPython 2012 25 / 40

Slide 40

Slide 40 text

Deferring your tasks Introducing celery Flask-Celery Flask-Script is a requirement Most of the commands work Except for starting detached celery daemons A. Mishkovskyi () Diving Into Flask EuroPython 2012 26 / 40

Slide 41

Slide 41 text

Deferring your tasks Introducing celery Flask-Celery from celery.platforms import detached class CeleryDetached (celeryd ): def run(self , ** kwargs ): sys.argv [1] = ’celeryd ’ with detached(kwargs[’logfile ’], kwargs[’pidfile ’]): os.execv(sys.argv [0], sys.argv) A. Mishkovskyi () Diving Into Flask EuroPython 2012 26 / 40

Slide 42

Slide 42 text

Deferring your tasks Celery and logging Color formatting Problem Celery always colorizes logs. We don’t like colors. A. Mishkovskyi () Diving Into Flask EuroPython 2012 27 / 40

Slide 43

Slide 43 text

Deferring your tasks Celery and logging OH HAI COLORZ A. Mishkovskyi () Diving Into Flask EuroPython 2012 27 / 40

Slide 44

Slide 44 text

Deferring your tasks Celery and logging Color formatting Problem Celery always colorizes logs. We don’t like colors. Solution Add after setup logger signal that reassigns all logging formatters for Celery logging handlers. A. Mishkovskyi () Diving Into Flask EuroPython 2012 27 / 40

Slide 45

Slide 45 text

Deferring your tasks Celery and logging Hijacking root logger Problem Root logger is hijacked by Celery’s logging setup, making your logging setup useless. Solution Set CELERYD HIJACK ROOT LOGGER to False. Or better yet, never use root logger. A. Mishkovskyi () Diving Into Flask EuroPython 2012 28 / 40

Slide 46

Slide 46 text

Deferring your tasks Celery and logging Process name Problem Logging might brake if you want to setup logging beyond log message format. See https://gist.github.com/721870 There are three places in the code where the processName is written to a LogRecord, some of which can lead to unexpected behaviour in some scenarios A. Mishkovskyi () Diving Into Flask EuroPython 2012 29 / 40

Slide 47

Slide 47 text

Deferring your tasks Celery and logging Process name Problem Logging might brake if you want to setup logging beyond log message format. Solution Avoid those scenarios. A. Mishkovskyi () Diving Into Flask EuroPython 2012 29 / 40

Slide 48

Slide 48 text

Deferring your tasks Monitoring Celery Keeping an eye on Celery Subclass celery.events.snapshot.Polaroid ??? PROFIT A. Mishkovskyi () Diving Into Flask EuroPython 2012 30 / 40

Slide 49

Slide 49 text

Deferring your tasks Monitoring Celery Keeping an eye on Celery Subclass celery.events.snapshot.Polaroid Implement on shutter method Check various metrics Generate report in whatever format you need A. Mishkovskyi () Diving Into Flask EuroPython 2012 30 / 40

Slide 50

Slide 50 text

Deferring your tasks Monitoring Celery Keeping an eye on Celery from celery.events.snapshot import Polaroid class Camera(Polaroid ): def on_shutter(self , state ): if not state.event_count: return print(’Workers: {}’.format( state.workers )) # Check state.tasks , # state.alive_workers , # etc A. Mishkovskyi () Diving Into Flask EuroPython 2012 30 / 40

Slide 51

Slide 51 text

Deferring your tasks Celery and Flask-SQLAlchemy Celery + SQLAlchemy + MySQL Problem Each time worker starts, infamous MySQL error is raised: OperationalError: (2006, ’MySQL server has gone away’) Solution Drop the whole connection (engine) pool at worker init. A. Mishkovskyi () Diving Into Flask EuroPython 2012 31 / 40

Slide 52

Slide 52 text

Deferring your tasks Celery and Flask-SQLAlchemy Celery + SQLAlchemy + MySQL from celery import signals from ignite.models import db def reset_connections (**_): db.session.bind.dispose () signals.worker_init.connect( reset_connections ) A. Mishkovskyi () Diving Into Flask EuroPython 2012 31 / 40

Slide 53

Slide 53 text

Deferring your tasks Celery and Flask-SQLAlchemy Celery + SQLAlchemy + MySQL Problem Session not closed if exception happens midway through transaction. Solution Close the session in task postrun signal. A. Mishkovskyi () Diving Into Flask EuroPython 2012 31 / 40

Slide 54

Slide 54 text

Deferring your tasks Celery and Flask-SQLAlchemy Celery + SQLAlchemy + MySQL from celery import signals from ignite.models import db def task_postrun_handler (**_): try: db.session.commit () finally: db.session.close () signals.task_postrun .connect( task_postrun_handler ) A. Mishkovskyi () Diving Into Flask EuroPython 2012 31 / 40

Slide 55

Slide 55 text

Deferring your tasks Celery and Flask-SQLAlchemy Celery + SQLAlchemy + MySQL Problem Session still not closed properly if db object loses app context. Worker hangs too if that happens. RuntimeError: application not registered on db instance and no application bound to current context Solution Close the session in task postrun signal but only if there was an exception. A. Mishkovskyi () Diving Into Flask EuroPython 2012 31 / 40

Slide 56

Slide 56 text

Deferring your tasks Celery and Flask-SQLAlchemy Celery + SQLAlchemy + MySQL from celery import signals from ignite.models import db def task_postrun_handler (**_): try: db.session.commit () except RuntimeError : pass except Exception: db.session.close () signals.task_postrun .connect( task_postrun_handler ) A. Mishkovskyi () Diving Into Flask EuroPython 2012 31 / 40

Slide 57

Slide 57 text

Caching & proﬁling Section 5 Caching & proﬁling A. Mishkovskyi () Diving Into Flask EuroPython 2012 32 / 40

Slide 58

Slide 58 text

Caching & proﬁling Caching Flask-Cache Plenty of caching decorators Otherwise – thin wrapper around werkzeug.contrib.cache A. Mishkovskyi () Diving Into Flask EuroPython 2012 33 / 40

Slide 59

Slide 59 text

Caching & proﬁling Caching Really thin wrapper def get(self , *args , ** kwargs ): "Proxy function for internal cache object." return self.cache.get(*args , ** kwargs) def set(self , *args , ** kwargs ): "Proxy function for internal cache object." self.cache.set(*args , ** kwargs) # Also add , delete , delete_many , etc. A. Mishkovskyi () Diving Into Flask EuroPython 2012 33 / 40

Slide 60

Slide 60 text

Caching & proﬁling Caching Meh... Wrote our own cache classes With namespace support And consistent hashing (based on libketama) Also ﬁxed and improved Python libredis wrapper A. Mishkovskyi () Diving Into Flask EuroPython 2012 34 / 40

Slide 61

Slide 61 text

Caching & proﬁling Proﬁling statsd Use python-statsd I have no more bullet points to add here So, there ... A picture of a cat instead! A. Mishkovskyi () Diving Into Flask EuroPython 2012 35 / 40

Slide 62

Slide 62 text

Caching & proﬁling Proﬁling statsd A. Mishkovskyi () Diving Into Flask EuroPython 2012 35 / 40

Slide 63

Slide 63 text

Caching & proﬁling Proﬁling statsd def setup_statsd(app): host = app.config[’STATSD_HOST ’] port = app.config[’STATSD_PORT ’] connection = statsd.Connection( host=host , port=port , sample_rate =0.1) app.statsd = statsd.Client( ’ignite ’, statsd_connection ) def statsd_metric(metric , duration=None ): counter = app.statsd.get_client( class_=statsd.Counter) counter.increment(metric) if duration is not None: timer = current_app.statsd.get_client( class_=statsd.Timer) timer.send(metric , duration) A. Mishkovskyi () Diving Into Flask EuroPython 2012 35 / 40

Slide 64

Slide 64 text

Caching & profiling Profiling Flask-DebugToolbar Direct port of Django’s debug toolbar Great at identifying bottlenecks We also added memory profiling (Pympler) Also: great example for blueprint-based plugin A. Mishkovskyi () Diving Into Flask EuroPython 2012 36 / 40

Slide 65

Slide 65 text

Conclusion Section 6 Conclusion A. Mishkovskyi () Diving Into Flask EuroPython 2012 37 / 40

Slide 66

Slide 66 text

Conclusion Flask maturity Flask is no longer an April Fool’s joke Still micro, but not in terms of features You can and should build applications with Flask Flask is easy to reason about A. Mishkovskyi () Diving Into Flask EuroPython 2012 38 / 40

Slide 67

Slide 67 text

Conclusion Flask’s ecosystem Not on par with Flask in places Interoperability is rough in places Lack’s BDFL for extensions (mitsuhiko for president!) A. Mishkovskyi () Diving Into Flask EuroPython 2012 39 / 40

Slide 68

Slide 68 text

Conclusion Questions? A. Mishkovskyi () Diving Into Flask EuroPython 2012 40 / 40