Upgrade to Pro — share decks privately, control downloads, hide ads and more …

asyncio - deep dive

asyncio - deep dive

Ca3d0556227d66b3c15be1eadf69473b?s=128

Jacob Tomlinson

July 11, 2017
Tweet

Transcript

  1. asyncio the exciting world of concurrent Python coroutines… Jacob Tomlinson

  2. https://opsdroid.github.io

  3. Concurrency and coroutines

  4. “Coroutines are computer program components that generalize subroutines for nonpreemptive

    multitasking, by allowing multiple entry points for suspending and resuming execution at certain locations” - Wikipedia "coroutines are functions whose execution you can pause"- Brett Cannon (Python core) -or-
  5. Isn’t that like threading?

  6. Common use cases • Event driven applications • IO bound

    applications • Long running IO events (websockets) • Applications which make lots of network requests (uploading/downloading data)
  7. Once upon a time...

  8. 1. Generators

  9. 1.1 yield (PEP 255, Python 2.2)

  10. 1.1 Generators - yield def eager_range(up_to): """Create a list of

    integers, from 0 to up_to, exclusive.""" sequence = [] index = 0 while index < up_to: sequence.append(index) index += 1 return sequence
  11. 1.1 Generators - yield def lazy_range(up_to): """Generator to return the

    sequence of integers from 0 to up_to, exclusive.""" index = 0 while index < up_to: yield index index += 1
  12. 1.2 value = yield (PEP 342, Python 2.5)

  13. 1.2 Generators - value = yield def jumping_range(up_to): """Generator for

    the sequence of integers from 0 to up_to, exclusive. Sending a value into the generator will shift the sequence by that amount. """ index = 0 while index < up_to: jump = yield index if jump is None: jump = 1 index += jump if __name__ == '__main__': iterator = jumping_range(5) print(next(iterator)) # 0 print(iterator.send(2)) # 2 print(next(iterator)) # 3 print(iterator.send(-1)) # 2 for x in iterator: print(x) # 3, 4
  14. 1.3 yield from (PEP 380, Python 3.3)

  15. 1.3 Generators - yield from yield from iterator # was

    roughly equivalent to for x in iterator: yield x
  16. 1.3 Generators - yield from def bottom(): # Returning the

    yield lets the value that goes up the call stack to come right back # down. return (yield 42) def middle(): return (yield from bottom()) def top(): return (yield from middle()) # Get the generator. gen = top() value = next(gen) print(value) # Prints '42'.
  17. 2. asyncio (Python 3.4)

  18. An event loop “is a programming construct that waits for

    and dispatches events or messages in a program” - Wikipedia
  19. 2.1 @asyncio.coroutine (Python 3.4)

  20. 2.1 asyncio - coroutine import time def countdown(number, n): while

    n > 0: print('T-minus', n, '({})'.format(number)) time.sleep(1) n -= 1 countdown("A", 2) countdown("B", 3) Output T-minus 2 (A) T-minus 1 (A) T-minus 3 (B) T-minus 2 (B) T-minus 1 (B) Time taken: 5s
  21. Demo 2.1 asyncio - coroutine

  22. 2.1 asyncio - coroutine import asyncio @asyncio.coroutine def countdown(number, n):

    while n > 0: print('T-minus', n, '({})'.format(number)) yield from asyncio.sleep(1) n -= 1 loop = asyncio.get_event_loop() tasks = [ asyncio.ensure_future(countdown("A", 2)), asyncio.ensure_future(countdown("B", 3))] loop.run_until_complete(asyncio.wait(tasks)) loop.close() Output T-minus 2 (A) T-minus 3 (B) T-minus 1 (A) T-minus 2 (B) T-minus 1 (B) Time taken: 3s
  23. Demo 2.1 asyncio - coroutine

  24. 2.2 Future and Task (Python 3.4)

  25. 4. aiohttp A Future is an object that represents the

    result of work that hasn’t completed. A Task is a wrapper for a coroutine and a subclass of Future.
  26. Job 1 Job 2 Job 1.1 Job 2.1 Job 1.1.1

  27. 3. async and await

  28. 3. async and await # Python 3.4 @asyncio.coroutine def py34_coro():

    yield from stuff() # Python 3.5 async def py35_coro(): await stuff()
  29. 4. Compatible libraries

  30. 4.1 aiohttp (PEP 3156) HTTP client/server for asyncio

  31. 4.1 aiohttp • Supports both Client and HTTP Server. •

    Supports both Server WebSockets and Client WebSockets out-of-the-box. • Web-server has Middlewares, Signals and pluggable routing.
  32. 4.1.1 Libraries - aiohttp - client import aiohttp import asyncio

    import async_timeout async def fetch(session, url): with async_timeout.timeout(10): async with session.get(url) as response: return await response.text() async def main(): async with aiohttp.ClientSession() as session: html = await fetch(session, 'http://python.org') print(html) loop = asyncio.get_event_loop() loop.run_until_complete(main())
  33. Demo 4.1.1 Libraries - aiohttp - client

  34. 4.1.2 Libraries - aiohttp - server from aiohttp import web

    async def handle(request): name = request.match_info.get('name', "Anonymous") text = "Hello, " + name return web.Response(text=text) app = web.Application() app.router.add_get('/', handle) app.router.add_get('/{name}', handle) web.run_app(app)
  35. 4.2 asynctest Unit tests for asyncio

  36. 4.3 database modules asyncpg, aiomysql, aioodbc, motor, asyncio-redis, aioes...

  37. 5. run_in_executor aka legacy support

  38. 5.1 run_in_executor - blocking import asyncio import time async def

    my_blocking_coroutine(): print("Blocking started") time.sleep(4) print("Blocking done") async def my_nonblocking_coroutine(): print("Nonblocking started") await asyncio.sleep(2) print("Nonblocking done") loop = asyncio.get_event_loop() tasks = [ asyncio.ensure_future(my_nonblocking_coroutine()), asyncio.ensure_future(my_blocking_coroutine())] loop.run_until_complete(asyncio.wait(tasks))
  39. Demo 5.1 run_in_executor - blocking

  40. 5.2 run_in_executor - nonblocking import asyncio import time async def

    my_blocking_coroutine(): print("Blocking started") await loop.run_in_executor(None, time.sleep, 4) print("Blocking done") async def my_nonblocking_coroutine(): print("Nonblocking started") await asyncio.sleep(2) print("Nonblocking done") loop = asyncio.get_event_loop() tasks = [ asyncio.ensure_future(my_nonblocking_coroutine()), asyncio.ensure_future(my_blocking_coroutine())] loop.run_until_complete(asyncio.wait(tasks))
  41. Demo 5.2 run_in_executor - blocking

  42. There’s even more! but we don’t have time for that

    (you may have even lost the will to live by now) • Queues • Gathers • Sockets • Signals • Pipes • Transports • Streams • Subprocesses • Exceptions • Protocols • more...
  43. Further Reading http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5 https://docs.python.org/3/library/asyncio.html http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio/ https://carlosmaniero.github.io/asyncio-handle-blocking-functions.html https://github.com/dabeaz/curio