Is asyncio stack ready for web development?

Slide 1

Slide 1 text

Is asyncio stack ready Is asyncio stack ready for web development? for web development? Igor Davydenko 2018, PyCon CZ

Slide 2

Slide 2 text

I am… I am… Igor Davydenko Python / JavaScript developer Making web applications for last 14 years 14 years Making them primarily on Python for last 10 years 10 years

Slide 3

Slide 3 text

Agenda, Agenda, kind of kind of My last to date Python project My last to date Python project Wri en on aioh p Works at production for 15+ months Project for signing and sharing legal documents in Ukraine Sign process done in JavaScript mostly All other necessary stuﬀ done in Python

Slide 4

Slide 4 text

Why to choose aioh p? Why to choose aioh p? Over Django / Flask? Over Django / Flask?

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

aioh p to be chosen aioh p to be chosen When sync frameworks fail When sync frameworks fail

Slide 7

Slide 7 text

Django request-response cycle Django request-response cycle

Slide 8

Slide 8 text

Some general problems with sync ﬂow Some general problems with sync ﬂow Serving big amount of concurrent users Handling big amount of connections with data sources Making big amount of requests to external sources

Slide 9

Slide 9 text

Solving problmes with sync ﬂow Solving problmes with sync ﬂow Scale horizontally Adding magic: eventlet / gevent Start looking for another solutions ... Switch to Golang

Slide 10

Slide 10 text

Another solutions Another solutions Python developers experienced same problems They want to bring a be er concurrency in Python asyncio was born asyncio start primarily using for web development

Slide 11

Slide 11 text

import asyncio import asyncio

Slide 12

Slide 12 text

asyncio asyncio Added to Python in 3.4 Just infrastructure for writing concurrent code Async I/O, event loop, coroutines, and tasks import asyncio async def hello_world() -> None: print('Hello, world!') asyncio.run(hello.world())

Slide 13

Slide 13 text

asyncio asyncio is just an infrasctructure is just an infrasctructure asyncio itself is not suitable for web development You need to: have web framework built on top of it: aiohttp.web communicate with data sources: aiopg, asyncpg, aioredis, aio* communicate with external API: aiohttp.client

Slide 14

Slide 14 text

asyncio asyncio stack provides stack provides Concurrent code Concurrent view execution Concurrent communication with data sources Concurrent fetching data from external API Deployment without application server async / await all around your code

Slide 15

Slide 15 text

This why we chose This why we chose aiohttp aiohttp A empt to use as low resources as possible A lot of concurrent requests from users A lot of data to receive (uploaded documents) from users A lot of data to send to external sources Communicate with external APIs

Slide 16

Slide 16 text

from aiohttp import web from aiohttp import web

Slide 17

Slide 17 text

aiohttp.web aiohttp.web is not a rocket science is not a rocket science Very similar to sync frameworks Init app, setup routes, run the app from aiohttp import web async def hello_world(request: web.Request) -> web.Response: return web.json_response({'data': 'Hello, world!'}) app = web.Application() app.router.add_get('/', hello_world) web.run_app(app)

Slide 18

Slide 18 text

Deploying without application server Deploying without application server yourapp/__main__.py yourapp/__main__.py import asyncio import sys import uvloop from aiohttp import web from yourapp.app import create_app if __name__ == '__main__': asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) app = create_app() sys.exit(web.run_app(app)) $ python -m yourapp

Slide 19

Slide 19 text

Ba eries not included Ba eries not included from aiopg.sa import create_engine async def connect_db(app: web.Application) -> None: app['db'] = await create_engine(app['config']['db']['url']) async def disconnect_db(app: web.Application) -> None: db = app.get('db') if db: db.close() await db.wait_closed() app.on_startup.append(connect_db) app.on_shutdown.append(disconnect_db)

Slide 20

Slide 20 text

Views are same old views Views are same old views This is not a production-ready code! This is not a production-ready code! async def upload_document(request: web.Request) -> web.Response: data = await request.post() files = {} for file in data.values(): files[file.filename] = file.body await upload_to_s3(file.filename, file.body) async with request.app['db'].acquire() as conn: for file in files: await insert_document(conn, file) return web.json_response(status=201)

Slide 21

Slide 21 text

REST API? REST API? REST API done manually Validate / transform request data with trafaret Process safe data Return JSON response (mostly empty one) For public API we're used Swagger for docs Maybe there is Django REST Framework for aioh p, but we didn't aware of

Slide 22

Slide 22 text

GraphQL? GraphQL? GraphQL built on top of hiku: Read only for fetching data Mutations not yet supported As well as many other neat features If you need more powerful, try graphql-aioh p

Slide 23

Slide 23 text

Other? Other? Tasks queue (Kafka) Internal API for communicating with other holding products And more…

Slide 24

Slide 24 text

Did we really need aioh p? Did we really need aioh p?

Slide 25

Slide 25 text

Developers are lazy Developers are lazy They wanted that framework X covered all cases But each tool / library may be good in one particular case Django is from good to great for newspaper site aioh p is not

Slide 26

Slide 26 text

Ba eries not included Ba eries not included No Django admin There is aioh p-admin, but c'mon. Same as Flask-Admin vs Django Admin No Django REST Framework No ORM

Slide 27

Slide 27 text

No ORM No ORM And you might don't need it query = ( sa.select([sa.func.max(document_file_table.c.content_length)]) .select_from( select_from .outerjoin( document_file_table, document_file_table.c.document_id == document_table.c.id, ) .outerjoin( document_meta_table, document_meta_table.c.id == document_file_table.c.meta_id, ), ) .where(sa.and_( clause, document_meta_table.c.is_current.is_(True), document_file_table.c.type == DocumentFileType.original, )) )

Slide 28

Slide 28 text

Evaluate before implement Evaluate before implement We didn't need admin for managing data We were fine with querying data without ORM We were fine with omiting REST API framework We chose aioh p cause of concurrency and let see how it payed off

Slide 29

Slide 29 text

Real life lessons Real life lessons From running aioh p in production From running aioh p in production

Slide 30

Slide 30 text

aioh p doesn't push your app structure aioh p doesn't push your app structure The freedom is great But for one dev project More devs -> your project became a mess You need to enforce not only coding style, but to conform on structure of your code as well

Slide 31

Slide 31 text

App structure App structure app subapp db.py enums.py tables.py types.py utils.py views.py validators.py __main__.py app.py conﬁg.py routes.py signals.py

Slide 32

Slide 32 text

Se ings Se ings The freedom. Part 2 :( You need to agree on how to manage se ings for your app by yourself Our setup: Store se ings in *.yaml Provide trafaret schema to validate se ings Use trafaret-conﬁg for reading se ings Error in se ings? App even didn't start

Slide 33

Slide 33 text

Expect everything Expect everything aioh p is a quite young project You need to expect everything 2 times our code broken a er aioh p update Still used old uvloop version due inability to upgrade to 0.9.1

Slide 34

Slide 34 text

Tests are the necessity Tests are the necessity With aioh p it becames more obvious Try to achieve 85% coverage We started with 5% and each deploy was … You also need to test routine things as well

Slide 35

Slide 35 text

Typing will save you as well Typing will save you as well Type annotations indirectly enforce you to write more understandable code Instead of dict you might want to start using namedtuple or dataclass, or at least StructedDict Documentation for your code Your teammates with IDE thank you everyday

Slide 36

Slide 36 text

CPU Bound code? Run it in executor CPU Bound code? Run it in executor The power of asyncio in await statement No await – asyncio is not your choice But there is run_in_executor async def gzip_buffer(buffer: io.BytesIO, *, loop: asyncio.AbstractEventLoop) -> io.BytesIO: """Non-blocking way of compressing prepared bytes IO with gzip.""" return await loop.run_in_executor(None, sync_gzip_buffer, buffer)

Slide 37

Slide 37 text

aioh p still in development aioh p still in development aiohttp.web sill in semi-active development New feature are coming for sure (like @route decorator) aio-* libs might not cover your case You might need to payback to open-source Like ﬁxing bugs, that blocks you from update to new aioh p version

Slide 38

Slide 38 text

web.Application web.Application is a context holder is a context holder It's just a dict You can use your app instance everywhere In web server context In tasks queue context

Slide 39

Slide 39 text

Where to find aioh p devs? Where to find aioh p devs? Your project grows – you need more devs Where to find them? The market offers much more Django / Flask devs, then aioh p devs Especially it is hard to substitute senior / lead dev

Slide 40

Slide 40 text

How to grow aioh p dev? How to grow aioh p dev? Start with basics: how asyncio works, tests, etc… Asyncio requires time for diving in Continue with basics: read & discuss aioh p code aiohttp is not a rocket science It may pay dividends later

Slide 41

Slide 41 text

More lessons More lessons From running aioh p app in production From running aioh p app in production

Slide 42

Slide 42 text

Logs are important Logs are important

Slide 43

Slide 43 text

Metrics are important Metrics are important

Slide 44

Slide 44 text

Dev env == prod env Dev env == prod env A empt to make dev env as close as possible to prod env vagga make containers for dev lithos run containers at staging / prod

Slide 45

Slide 45 text

Proﬁle your app Proﬁle your app import cProfile profiler = cProfile.Profile() if __name__ == '__main__': use_profiler = os.environ.get('USE_PROFILER') == '1' if use_profiler: profiler.enable() app = create_app() try: sys.exit(web.run_app(app)) finally: if use_profiler: profiler.dump_stats('yourapp.prof') profiler.disable()

Slide 46

Slide 46 text

Proﬁle your app Proﬁle your app

Slide 47

Slide 47 text

Debug your app Debug your app aiomonitor

Slide 48

Slide 48 text

What's next for asyncio? What's next for asyncio?

Slide 49

Slide 49 text

What's new in Python 3.7 What's new in Python 3.7 asyncio.run contextvars support More performance improvements

Slide 50

Slide 50 text

Competitors Competitors As async / await is part of Python, other devs able to make other async libraries curio trio Maybe we are close to curhttp or trhttp :)

Slide 51

Slide 51 text

Does Python became be er a er asyncio? Does Python became be er a er asyncio? Yes! Yes!

Slide 52

Slide 52 text

Was aioh p choice worth it? Was aioh p choice worth it? Yes! Yes!

Slide 53

Slide 53 text

Did I repeat a ride? Did I repeat a ride? I'm not sure I'm not sure

Slide 54

Slide 54 text

asyncio stack is harder asyncio stack is harder Still new technology Less developers involved More challenges ahead of you

Slide 55

Slide 55 text

But, asyncio stack is ready But, asyncio stack is ready Choose wisely Expect everything With more challenges you became a be er dev Join the ride!

Slide 56

Slide 56 text

Questions? Questions? Twi er: Twi er: @playpausenstop @playpausenstop GitHub: GitHub: @playpauseandstop @playpauseandstop Made in Ukraine