You Might Not Want Async

You Might Not Want Async

First version given at PyCon TW 2016.
Revised for PyCon APAC 2016 (Seoul).
Revised for PyCon JP 2016.

Asynchrony in Python had gathered much momentum recently, with interests from core developers, as evidenced by the introduction of `asyncio` in Python 3.4, and a great boom of related third-party projects following it. By utilising more functionalities from the underlying operating system, it is a great solution to many existing problems in Python applications, gaining practical concurrency without working around the well-known GIL (global interpreter lock) problem.

With all its advantages, asynchrony is, however, still a relatively new concept in Python, and as a result could be somewhat mistaken, even misunderstood by some people. One of these misconceptions, probably the most serious, is to mistake concurrency through asynchrony for parallelism. Although `asyncio` (and other similar solutions) lets multiple parts of your program executes without interfering each other, it does *not* allow them to run together—this is still impossible, at least in CPython, due to the continued existence of the GIL. This makes asynchrony suitable for only a certain, instead of all, kinds of problems. Evaluation is therefore required before a programmer can decide whether the asynchrony model is suitable for a particular application.

Furthermore, partly due to its relatively short existence, paradigms in asynchrony programming do not necessarily fit well with other parts of Python, including libraries, either built-in or third-party ones. Since only blocking libraries were available in most of Python’s history, many assumptions they made may not work well with async programs out-of-the-box. Adopting asynchrony, at least at the present time, will therefore introduce more technical debt to your program. These are all important aspects that require much consideration before you dive head-first into asynchrony.

9dafad54b5b4f360b7aae5f482bc1c91?s=128

Tzu-ping Chung

June 04, 2016
Tweet

Transcript

  1. 5.
  2. 6.
  3. 10.
  4. 11.
  5. 12.

    meh

  6. 14.
  7. 16.
  8. 23.

    Not For You • Infects the whole program • Async

    is not parallelism • Third party support
  9. 24.

    import sqlite3 def read_data(dbname): con = sqlite3.connect(dbname) cur = con.cursor()

    cur.execute('SELECT * FROM data OFFSET 0 LIMIT 1') data = cur.fetchone() cur.close() con.close() return data
  10. 25.

    import aioodbc async def read_data(dbname): con = await aioodbc.connect( dsn='Driver=SQLite;Database={}'.format(dbname),

    ) cur = await con.cursor() await cur.execute('SELECT * FROM data OFFSET 0 LIMIT 1') data = await cur.fetchone() await cur.close() await con.close() return data
  11. 27.

    import asyncio from .db import read_data async def main(): data

    = await read_data('db.sqlite3') print(data) loop = asyncio.get_event_loop() loop.run_until_complete(main()) loop.close()
  12. 30.

    import asyncio from .db import read_data async def main(): data

    = read_data('db.sqlite3') print(data) loop = asyncio.get_event_loop() loop.run_until_complete(main()) loop.close()
  13. 32.

    import asyncio from .db import read_data async def main(): data

    = read_data('db.sqlite3') print(data) loop = asyncio.get_event_loop() loop.run_until_complete(main()) loop.close()
  14. 33.

    import asyncio from .db import read_data async def main(): data

    = await read_data('db.sqlite3') print(data) loop = asyncio.get_event_loop() loop.run_until_complete(main()) loop.close()
  15. 34.
  16. 37.
  17. 38.
  18. 39.

    import asyncio import time async def do_something(): print('Before', time.monotonic()) await

    asyncio.sleep(2) print('After ', time.monotonic())  await do_something() Before 199208.190632853 After 199210.192092425
  19. 43.

    What Happened? • unittest does not know about asyncio •

    Coroutine methods are executed “normally” • Called, but not executed (awaited)
  20. 45.

    import asyncio import functools def asynchronous(func): @functools.wraps(func) def asynchronous_inner(*args, **kwargs):

    loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) try: loop.run_until_complete(func(*args, **kwargs)) finally: loop.close() return asynchronous_inner
  21. 48.

    Tips • Beware of warning output • Especially if you

    redirect • Add coverage report to testing code • Consider pip install asynctest
  22. 50.

    import time async def test_do_something(): before = time.monotonic() await do_something()

    delta_t = time.monotonic() - before assert -0.01 < delta_t < 0.01
  23. 51.

    (demo) $ py.test tests.py ====================== test session starts ====================== platform

    darwin -- Python 3.5.1, pytest-2.9.1, py-1.4.31, pluggy-0.3.1 collected 0 items / 1 errors ============================ ERRORS ============================= ___________________ ERROR collecting tests.py____________________ > for i, x in enumerate(self.obj()): E TypeError: 'coroutine' object is not iterable python3.5/site-packages/_pytest/python.py:765: TypeError ==================== 1 error in 0.15 seconds ====================
  24. 52.
  25. 53.

    import time import pytest @pytest.mark.asyncio async def test_do_something(capsys): before =

    time.monotonic() await do_something() delta_t = time.monotonic() - before assert -0.01 < delta_t < 0.01
  26. 54.
  27. 55.
  28. 56.
  29. 58.

    import requests def collect_contents(urls): contents = [] for url in

    urls: resp = requests.get(url) if resp.status_code != 200: continue content = resp.text contents.append(content) return contents
  30. 59.

    import aiohttp async def collect_contents(urls): contents = [] with aiohttp.ClientSession()

    as session: for url in urls: async with session.get(url) as rest: if resp.status != 200: continue content = await resp.text() contents.append(content) return contents
  31. 61.

    import aiohttp async def collect_contents(urls): contents = [] with aiohttp.ClientSession()

    as session: for url in urls: async with session.get(url) as rest: if resp.status != 200: continue content = await resp.text() contents.append(content) return contents
  32. 62.
  33. 63.

    import asyncio import aiohttp async def collect_contents(urls): coroutines = []

    with aiohttp.ClientSession() as session: for url in urls: async with session.get(url) as rest: if resp.status != 200: continue coroutines.append(resp.text()) contents = await asyncio.gather(*coroutines) return contents
  34. 64.
  35. 65.
  36. 66.
  37. 68.

    Alternatives • concurrent & multiprocessing • Greenlets • Similar idea,

    but less infectious • C extension • threading • Standard I/O
  38. 72.

    package main import ("fmt"; "time") func doSomething() { time.Sleep(2 *

    time.Second) fmt.Println(time.Now(), "Slept") } func main() { doSomething() fmt.Println(time.Now(), "OK") } 00:00:00 Slept 00:00:00 OK
  39. 73.

    package main import ("fmt"; "time") func doSomething() { time.Sleep(2 *

    time.Second) fmt.Println(time.Now(), "Slept") } func main() { go doSomething() fmt.Println(time.Now(), "OK") } 00:00:00 OK
  40. 74.

    package main import ("fmt"; "time") var sem = make(chan bool)

    func doSomething() { time.Sleep(2 * time.Second) fmt.Println(time.Now(), "Slept") sem  true } func main() { go doSomething() fmt.Println(time.Now(), "OK") sem } 00:00:00 OK 00:00:02 Slept
  41. 76.

    """Do something. Synchronous version.""" import datetime import time def do_something():

    time.sleep(2) print(datetime.datetime.now(), 'Slept') do_something() print(datetime.datetime.now(), 'Done')
  42. 77.

    """Do something. Asynchronous version.""" import asyncio import datetime async def

    do_something(): await asyncio.sleep(2) print(datetime.datetime.now(), 'Slept') loop = asyncio.get_event_loop() task = loop.create_task(do_something()) print(datetime.datetime.now(), 'Done') loop.run_until_complete(asyncio.wait([task])) loop.close()
  43. 78.
  44. 79.

    """Do something. Synchronous version.""" import datetime import time def do_something():

    time.sleep(2) print(datetime.datetime.now(), 'Slept') do_something() print(datetime.datetime.now(), 'Done')
  45. 80.

    """What if I can just write this?""" import asyncio import

    datetime import time async def do_something(): await time.sleep(2) print(datetime.datetime.now(), 'Slept') await do_something() print(datetime.datetime.now(), 'Done') asyncio.run_event_loop()
  46. 84.

    But Seriously • Asynchrony is not the silver bullet •

    It makes you jump through loops • There are alternatives • Fingers crossed
  47. 85.