Is asyncio stack ready for web development?

Is asyncio stack ready for web development?

Lessons learned from running aiohttp web application in production for 15+ months. Reasons why to choose asyncio stack for web development over standard Django / Flask approaches.

D809571c81f596ccc197f90142c966ab?s=128

Igor Davydenko

June 02, 2018
Tweet

Transcript

  1. 1.

    Is asyncio stack ready Is asyncio stack ready for web

    development? for web development? Igor Davydenko 2018, PyCon CZ
  2. 2.

    I am… I am… Igor Davydenko Python / JavaScript developer

    Making web applications for last 14 years 14 years Making them primarily on Python for last 10 years 10 years
  3. 3.

    Agenda, Agenda, kind of kind of My last to date

    Python project My last to date Python project Wri en on aioh p Works at production for 15+ months Project for signing and sharing legal documents in Ukraine Sign process done in JavaScript mostly All other necessary stuff done in Python
  4. 4.

    Why to choose aioh p? Why to choose aioh p?

    Over Django / Flask? Over Django / Flask?
  5. 5.
  6. 6.

    aioh p to be chosen aioh p to be chosen

    When sync frameworks fail When sync frameworks fail
  7. 8.

    Some general problems with sync flow Some general problems with

    sync flow Serving big amount of concurrent users Handling big amount of connections with data sources Making big amount of requests to external sources
  8. 9.

    Solving problmes with sync flow Solving problmes with sync flow

    Scale horizontally Adding magic: eventlet / gevent Start looking for another solutions ... Switch to Golang
  9. 10.

    Another solutions Another solutions Python developers experienced same problems They

    want to bring a be er concurrency in Python asyncio was born asyncio start primarily using for web development
  10. 12.

    asyncio asyncio Added to Python in 3.4 Just infrastructure for

    writing concurrent code Async I/O, event loop, coroutines, and tasks import asyncio async def hello_world() -> None: print('Hello, world!') asyncio.run(hello.world())
  11. 13.

    asyncio asyncio is just an infrasctructure is just an infrasctructure

    asyncio itself is not suitable for web development You need to: have web framework built on top of it: aiohttp.web communicate with data sources: aiopg, asyncpg, aioredis, aio* communicate with external API: aiohttp.client
  12. 14.

    asyncio asyncio stack provides stack provides Concurrent code Concurrent view

    execution Concurrent communication with data sources Concurrent fetching data from external API Deployment without application server async / await all around your code
  13. 15.

    This why we chose This why we chose aiohttp aiohttp

    A empt to use as low resources as possible A lot of concurrent requests from users A lot of data to receive (uploaded documents) from users A lot of data to send to external sources Communicate with external APIs
  14. 17.

    aiohttp.web aiohttp.web is not a rocket science is not a

    rocket science Very similar to sync frameworks Init app, setup routes, run the app from aiohttp import web async def hello_world(request: web.Request) -> web.Response: return web.json_response({'data': 'Hello, world!'}) app = web.Application() app.router.add_get('/', hello_world) web.run_app(app)
  15. 18.

    Deploying without application server Deploying without application server yourapp/__main__.py yourapp/__main__.py

    import asyncio import sys import uvloop from aiohttp import web from yourapp.app import create_app if __name__ == '__main__': asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) app = create_app() sys.exit(web.run_app(app)) $ python -m yourapp
  16. 19.

    Ba eries not included Ba eries not included from aiopg.sa

    import create_engine async def connect_db(app: web.Application) -> None: app['db'] = await create_engine(app['config']['db']['url']) async def disconnect_db(app: web.Application) -> None: db = app.get('db') if db: db.close() await db.wait_closed() app.on_startup.append(connect_db) app.on_shutdown.append(disconnect_db)
  17. 20.

    Views are same old views Views are same old views

    This is not a production-ready code! This is not a production-ready code! async def upload_document(request: web.Request) -> web.Response: data = await request.post() files = {} for file in data.values(): files[file.filename] = file.body await upload_to_s3(file.filename, file.body) async with request.app['db'].acquire() as conn: for file in files: await insert_document(conn, file) return web.json_response(status=201)
  18. 21.

    REST API? REST API? REST API done manually Validate /

    transform request data with trafaret Process safe data Return JSON response (mostly empty one) For public API we're used Swagger for docs Maybe there is Django REST Framework for aioh p, but we didn't aware of
  19. 22.

    GraphQL? GraphQL? GraphQL built on top of hiku: Read only

    for fetching data Mutations not yet supported As well as many other neat features If you need more powerful, try graphql-aioh p
  20. 25.

    Developers are lazy Developers are lazy They wanted that framework

    X covered all cases But each tool / library may be good in one particular case Django is from good to great for newspaper site aioh p is not
  21. 26.

    Ba eries not included Ba eries not included No Django

    admin There is aioh p-admin, but c'mon. Same as Flask-Admin vs Django Admin No Django REST Framework No ORM
  22. 27.

    No ORM No ORM And you might don't need it

    query = ( sa.select([sa.func.max(document_file_table.c.content_length)]) .select_from( select_from .outerjoin( document_file_table, document_file_table.c.document_id == document_table.c.id, ) .outerjoin( document_meta_table, document_meta_table.c.id == document_file_table.c.meta_id, ), ) .where(sa.and_( clause, document_meta_table.c.is_current.is_(True), document_file_table.c.type == DocumentFileType.original, )) )
  23. 28.

    Evaluate before implement Evaluate before implement We didn't need admin

    for managing data We were fine with querying data without ORM We were fine with omiting REST API framework We chose aioh p cause of concurrency and let see how it payed off
  24. 29.

    Real life lessons Real life lessons From running aioh p

    in production From running aioh p in production
  25. 30.

    aioh p doesn't push your app structure aioh p doesn't

    push your app structure The freedom is great But for one dev project More devs -> your project became a mess You need to enforce not only coding style, but to conform on structure of your code as well
  26. 31.

    App structure App structure app subapp db.py enums.py tables.py types.py

    utils.py views.py validators.py __main__.py app.py config.py routes.py signals.py
  27. 32.

    Se ings Se ings The freedom. Part 2 :( You

    need to agree on how to manage se ings for your app by yourself Our setup: Store se ings in *.yaml Provide trafaret schema to validate se ings Use trafaret-config for reading se ings Error in se ings? App even didn't start
  28. 33.

    Expect everything Expect everything aioh p is a quite young

    project You need to expect everything 2 times our code broken a er aioh p update Still used old uvloop version due inability to upgrade to 0.9.1
  29. 34.

    Tests are the necessity Tests are the necessity With aioh

    p it becames more obvious Try to achieve 85% coverage We started with 5% and each deploy was … You also need to test routine things as well
  30. 35.

    Typing will save you as well Typing will save you

    as well Type annotations indirectly enforce you to write more understandable code Instead of dict you might want to start using namedtuple or dataclass, or at least StructedDict Documentation for your code Your teammates with IDE thank you everyday
  31. 36.

    CPU Bound code? Run it in executor CPU Bound code?

    Run it in executor The power of asyncio in await statement No await – asyncio is not your choice But there is run_in_executor async def gzip_buffer(buffer: io.BytesIO, *, loop: asyncio.AbstractEventLoop) -> io.BytesIO: """Non-blocking way of compressing prepared bytes IO with gzip.""" return await loop.run_in_executor(None, sync_gzip_buffer, buffer)
  32. 37.

    aioh p still in development aioh p still in development

    aiohttp.web sill in semi-active development New feature are coming for sure (like @route decorator) aio-* libs might not cover your case You might need to payback to open-source Like fixing bugs, that blocks you from update to new aioh p version
  33. 38.

    web.Application web.Application is a context holder is a context holder

    It's just a dict You can use your app instance everywhere In web server context In tasks queue context
  34. 39.

    Where to find aioh p devs? Where to find aioh

    p devs? Your project grows – you need more devs Where to find them? The market offers much more Django / Flask devs, then aioh p devs Especially it is hard to substitute senior / lead dev
  35. 40.

    How to grow aioh p dev? How to grow aioh

    p dev? Start with basics: how asyncio works, tests, etc… Asyncio requires time for diving in Continue with basics: read & discuss aioh p code aiohttp is not a rocket science It may pay dividends later
  36. 41.

    More lessons More lessons From running aioh p app in

    production From running aioh p app in production
  37. 44.

    Dev env == prod env Dev env == prod env

    A empt to make dev env as close as possible to prod env vagga make containers for dev lithos run containers at staging / prod
  38. 45.

    Profile your app Profile your app import cProfile profiler =

    cProfile.Profile() if __name__ == '__main__': use_profiler = os.environ.get('USE_PROFILER') == '1' if use_profiler: profiler.enable() app = create_app() try: sys.exit(web.run_app(app)) finally: if use_profiler: profiler.dump_stats('yourapp.prof') profiler.disable()
  39. 49.

    What's new in Python 3.7 What's new in Python 3.7

    asyncio.run contextvars support More performance improvements
  40. 50.

    Competitors Competitors As async / await is part of Python,

    other devs able to make other async libraries curio trio Maybe we are close to curhttp or trhttp :)
  41. 51.

    Does Python became be er a er asyncio? Does Python

    became be er a er asyncio? Yes! Yes!
  42. 53.

    Did I repeat a ride? Did I repeat a ride?

    I'm not sure I'm not sure
  43. 54.

    asyncio stack is harder asyncio stack is harder Still new

    technology Less developers involved More challenges ahead of you
  44. 55.

    But, asyncio stack is ready But, asyncio stack is ready

    Choose wisely Expect everything With more challenges you became a be er dev Join the ride!
  45. 56.