$30 off During Our Annual Pro Sale. View Details »

asyncio - deep dive

asyncio - deep dive

Jacob Tomlinson

July 11, 2017
Tweet

More Decks by Jacob Tomlinson

Other Decks in Technology

Transcript

  1. “Coroutines are computer program components that generalize subroutines for nonpreemptive

    multitasking, by allowing multiple entry points for suspending and resuming execution at certain locations” - Wikipedia "coroutines are functions whose execution you can pause"- Brett Cannon (Python core) -or-
  2. Common use cases • Event driven applications • IO bound

    applications • Long running IO events (websockets) • Applications which make lots of network requests (uploading/downloading data)
  3. 1.1 Generators - yield def eager_range(up_to): """Create a list of

    integers, from 0 to up_to, exclusive.""" sequence = [] index = 0 while index < up_to: sequence.append(index) index += 1 return sequence
  4. 1.1 Generators - yield def lazy_range(up_to): """Generator to return the

    sequence of integers from 0 to up_to, exclusive.""" index = 0 while index < up_to: yield index index += 1
  5. 1.2 Generators - value = yield def jumping_range(up_to): """Generator for

    the sequence of integers from 0 to up_to, exclusive. Sending a value into the generator will shift the sequence by that amount. """ index = 0 while index < up_to: jump = yield index if jump is None: jump = 1 index += jump if __name__ == '__main__': iterator = jumping_range(5) print(next(iterator)) # 0 print(iterator.send(2)) # 2 print(next(iterator)) # 3 print(iterator.send(-1)) # 2 for x in iterator: print(x) # 3, 4
  6. 1.3 Generators - yield from yield from iterator # was

    roughly equivalent to for x in iterator: yield x
  7. 1.3 Generators - yield from def bottom(): # Returning the

    yield lets the value that goes up the call stack to come right back # down. return (yield 42) def middle(): return (yield from bottom()) def top(): return (yield from middle()) # Get the generator. gen = top() value = next(gen) print(value) # Prints '42'.
  8. An event loop “is a programming construct that waits for

    and dispatches events or messages in a program” - Wikipedia
  9. 2.1 asyncio - coroutine import time def countdown(number, n): while

    n > 0: print('T-minus', n, '({})'.format(number)) time.sleep(1) n -= 1 countdown("A", 2) countdown("B", 3) Output T-minus 2 (A) T-minus 1 (A) T-minus 3 (B) T-minus 2 (B) T-minus 1 (B) Time taken: 5s
  10. 2.1 asyncio - coroutine import asyncio @asyncio.coroutine def countdown(number, n):

    while n > 0: print('T-minus', n, '({})'.format(number)) yield from asyncio.sleep(1) n -= 1 loop = asyncio.get_event_loop() tasks = [ asyncio.ensure_future(countdown("A", 2)), asyncio.ensure_future(countdown("B", 3))] loop.run_until_complete(asyncio.wait(tasks)) loop.close() Output T-minus 2 (A) T-minus 3 (B) T-minus 1 (A) T-minus 2 (B) T-minus 1 (B) Time taken: 3s
  11. 4. aiohttp A Future is an object that represents the

    result of work that hasn’t completed. A Task is a wrapper for a coroutine and a subclass of Future.
  12. 3. async and await # Python 3.4 @asyncio.coroutine def py34_coro():

    yield from stuff() # Python 3.5 async def py35_coro(): await stuff()
  13. 4.1 aiohttp • Supports both Client and HTTP Server. •

    Supports both Server WebSockets and Client WebSockets out-of-the-box. • Web-server has Middlewares, Signals and pluggable routing.
  14. 4.1.1 Libraries - aiohttp - client import aiohttp import asyncio

    import async_timeout async def fetch(session, url): with async_timeout.timeout(10): async with session.get(url) as response: return await response.text() async def main(): async with aiohttp.ClientSession() as session: html = await fetch(session, 'http://python.org') print(html) loop = asyncio.get_event_loop() loop.run_until_complete(main())
  15. 4.1.2 Libraries - aiohttp - server from aiohttp import web

    async def handle(request): name = request.match_info.get('name', "Anonymous") text = "Hello, " + name return web.Response(text=text) app = web.Application() app.router.add_get('/', handle) app.router.add_get('/{name}', handle) web.run_app(app)
  16. 5.1 run_in_executor - blocking import asyncio import time async def

    my_blocking_coroutine(): print("Blocking started") time.sleep(4) print("Blocking done") async def my_nonblocking_coroutine(): print("Nonblocking started") await asyncio.sleep(2) print("Nonblocking done") loop = asyncio.get_event_loop() tasks = [ asyncio.ensure_future(my_nonblocking_coroutine()), asyncio.ensure_future(my_blocking_coroutine())] loop.run_until_complete(asyncio.wait(tasks))
  17. 5.2 run_in_executor - nonblocking import asyncio import time async def

    my_blocking_coroutine(): print("Blocking started") await loop.run_in_executor(None, time.sleep, 4) print("Blocking done") async def my_nonblocking_coroutine(): print("Nonblocking started") await asyncio.sleep(2) print("Nonblocking done") loop = asyncio.get_event_loop() tasks = [ asyncio.ensure_future(my_nonblocking_coroutine()), asyncio.ensure_future(my_blocking_coroutine())] loop.run_until_complete(asyncio.wait(tasks))
  18. There’s even more! but we don’t have time for that

    (you may have even lost the will to live by now) • Queues • Gathers • Sockets • Signals • Pipes • Transports • Streams • Subprocesses • Exceptions • Protocols • more...