Upgrade to Pro — share decks privately, control downloads, hide ads and more …

asyncio - deep dive

asyncio - deep dive

Jacob Tomlinson

July 11, 2017
Tweet

More Decks by Jacob Tomlinson

Other Decks in Technology

Transcript

  1. asyncio the exciting world of concurrent Python coroutines… Jacob Tomlinson

  2. https://opsdroid.github.io

  3. Concurrency and coroutines

  4. “Coroutines are computer program components that generalize subroutines for nonpreemptive

    multitasking, by allowing multiple entry points for suspending and resuming execution at certain locations” - Wikipedia "coroutines are functions whose execution you can pause"- Brett Cannon (Python core) -or-
  5. Isn’t that like threading?

  6. Common use cases • Event driven applications • IO bound

    applications • Long running IO events (websockets) • Applications which make lots of network requests (uploading/downloading data)
  7. Once upon a time...

  8. 1. Generators

  9. 1.1 yield (PEP 255, Python 2.2)

  10. 1.1 Generators - yield def eager_range(up_to): """Create a list of

    integers, from 0 to up_to, exclusive.""" sequence = [] index = 0 while index < up_to: sequence.append(index) index += 1 return sequence
  11. 1.1 Generators - yield def lazy_range(up_to): """Generator to return the

    sequence of integers from 0 to up_to, exclusive.""" index = 0 while index < up_to: yield index index += 1
  12. 1.2 value = yield (PEP 342, Python 2.5)

  13. 1.2 Generators - value = yield def jumping_range(up_to): """Generator for

    the sequence of integers from 0 to up_to, exclusive. Sending a value into the generator will shift the sequence by that amount. """ index = 0 while index < up_to: jump = yield index if jump is None: jump = 1 index += jump if __name__ == '__main__': iterator = jumping_range(5) print(next(iterator)) # 0 print(iterator.send(2)) # 2 print(next(iterator)) # 3 print(iterator.send(-1)) # 2 for x in iterator: print(x) # 3, 4
  14. 1.3 yield from (PEP 380, Python 3.3)

  15. 1.3 Generators - yield from yield from iterator # was

    roughly equivalent to for x in iterator: yield x
  16. 1.3 Generators - yield from def bottom(): # Returning the

    yield lets the value that goes up the call stack to come right back # down. return (yield 42) def middle(): return (yield from bottom()) def top(): return (yield from middle()) # Get the generator. gen = top() value = next(gen) print(value) # Prints '42'.
  17. 2. asyncio (Python 3.4)

  18. An event loop “is a programming construct that waits for

    and dispatches events or messages in a program” - Wikipedia
  19. 2.1 @asyncio.coroutine (Python 3.4)

  20. 2.1 asyncio - coroutine import time def countdown(number, n): while

    n > 0: print('T-minus', n, '({})'.format(number)) time.sleep(1) n -= 1 countdown("A", 2) countdown("B", 3) Output T-minus 2 (A) T-minus 1 (A) T-minus 3 (B) T-minus 2 (B) T-minus 1 (B) Time taken: 5s
  21. Demo 2.1 asyncio - coroutine

  22. 2.1 asyncio - coroutine import asyncio @asyncio.coroutine def countdown(number, n):

    while n > 0: print('T-minus', n, '({})'.format(number)) yield from asyncio.sleep(1) n -= 1 loop = asyncio.get_event_loop() tasks = [ asyncio.ensure_future(countdown("A", 2)), asyncio.ensure_future(countdown("B", 3))] loop.run_until_complete(asyncio.wait(tasks)) loop.close() Output T-minus 2 (A) T-minus 3 (B) T-minus 1 (A) T-minus 2 (B) T-minus 1 (B) Time taken: 3s
  23. Demo 2.1 asyncio - coroutine

  24. 2.2 Future and Task (Python 3.4)

  25. 4. aiohttp A Future is an object that represents the

    result of work that hasn’t completed. A Task is a wrapper for a coroutine and a subclass of Future.
  26. Job 1 Job 2 Job 1.1 Job 2.1 Job 1.1.1

  27. 3. async and await

  28. 3. async and await # Python 3.4 @asyncio.coroutine def py34_coro():

    yield from stuff() # Python 3.5 async def py35_coro(): await stuff()
  29. 4. Compatible libraries

  30. 4.1 aiohttp (PEP 3156) HTTP client/server for asyncio

  31. 4.1 aiohttp • Supports both Client and HTTP Server. •

    Supports both Server WebSockets and Client WebSockets out-of-the-box. • Web-server has Middlewares, Signals and pluggable routing.
  32. 4.1.1 Libraries - aiohttp - client import aiohttp import asyncio

    import async_timeout async def fetch(session, url): with async_timeout.timeout(10): async with session.get(url) as response: return await response.text() async def main(): async with aiohttp.ClientSession() as session: html = await fetch(session, 'http://python.org') print(html) loop = asyncio.get_event_loop() loop.run_until_complete(main())
  33. Demo 4.1.1 Libraries - aiohttp - client

  34. 4.1.2 Libraries - aiohttp - server from aiohttp import web

    async def handle(request): name = request.match_info.get('name', "Anonymous") text = "Hello, " + name return web.Response(text=text) app = web.Application() app.router.add_get('/', handle) app.router.add_get('/{name}', handle) web.run_app(app)
  35. 4.2 asynctest Unit tests for asyncio

  36. 4.3 database modules asyncpg, aiomysql, aioodbc, motor, asyncio-redis, aioes...

  37. 5. run_in_executor aka legacy support

  38. 5.1 run_in_executor - blocking import asyncio import time async def

    my_blocking_coroutine(): print("Blocking started") time.sleep(4) print("Blocking done") async def my_nonblocking_coroutine(): print("Nonblocking started") await asyncio.sleep(2) print("Nonblocking done") loop = asyncio.get_event_loop() tasks = [ asyncio.ensure_future(my_nonblocking_coroutine()), asyncio.ensure_future(my_blocking_coroutine())] loop.run_until_complete(asyncio.wait(tasks))
  39. Demo 5.1 run_in_executor - blocking

  40. 5.2 run_in_executor - nonblocking import asyncio import time async def

    my_blocking_coroutine(): print("Blocking started") await loop.run_in_executor(None, time.sleep, 4) print("Blocking done") async def my_nonblocking_coroutine(): print("Nonblocking started") await asyncio.sleep(2) print("Nonblocking done") loop = asyncio.get_event_loop() tasks = [ asyncio.ensure_future(my_nonblocking_coroutine()), asyncio.ensure_future(my_blocking_coroutine())] loop.run_until_complete(asyncio.wait(tasks))
  41. Demo 5.2 run_in_executor - blocking

  42. There’s even more! but we don’t have time for that

    (you may have even lost the will to live by now) • Queues • Gathers • Sockets • Signals • Pipes • Transports • Streams • Subprocesses • Exceptions • Protocols • more...
  43. Further Reading http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5 https://docs.python.org/3/library/asyncio.html http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio/ https://carlosmaniero.github.io/asyncio-handle-blocking-functions.html https://github.com/dabeaz/curio