Python and Asynchrony

Presented at Jobspace on 14 October, 2016.

An introduction to asynchrony programming in Python. This presentation consists of two parts: First I introduce the idea behind asynchrony, and how it differs from more “traditional” (in Python sense) concurrent solutions.

In the second part I use code examples to walk through the development of async paradigms, introducing async through callbacks, promises, and finally coroutine, so we get a deeper understanding how the underlying plumbing works when you run an async program.

Code examples for the second part of the presentation can be found at http://bit.ly/asyncio-examples.

Transcript

1. Python and Asynchrony
   

   View Slide

2. http://bit.ly/asyncio-examples
   

   View Slide

3. Quick Questions
   • Concurrency with Python
   • Threads and multi-processing
   • Single-thread asynchrony
   • asyncio, Twisted, Tornado, etc.
   

   View Slide

4. Me
   • Call me TP
   • Follow @uranusjr
   • https://uranusjr.com
   

   View Slide

5. View Slide

6. View Slide

7. http://macdown.uranusjr.com
   

   View Slide

8. www. .com
   

   View Slide

9. 10–11 June 2017 (Ծ)
   7
   

   View Slide

10. View Slide

11. Agenda
    • Introduction to asynchrony
    • Async the asyncio way
    • The road ahead
    

    View Slide

12. View Slide

13. Waiting (wasted)
    Synchronous
    

    View Slide

14. Threads
    

    View Slide

15. Threads
    • More than you need
    • Race conditions
    • Dead-locking
    • Resource starvation
    

    View Slide

16. To share, or not to share:
    that is the question.
    

    View Slide

17. LET’S TALK ABOUT SOMETHING
    “DIFFERENT”
    

    View Slide

18. Context Switching
    • First as a CPU thing
    • Interrupts
    • Software interrupts
    

    View Slide

19. Multitasking
    • A common OS problem
    • More tasks than threads
    • Cooperative
    • Pre-emptive
    

    View Slide

20. View Slide

21. Same
    problem?
    

    View Slide

22. Threads
    

    View Slide

23. Scheduling
    

    View Slide

24. Asynchronous
    

    View Slide

25. Sync vs. Async
    

    View Slide

26. ASYNCIO
    

    View Slide

27. Async How?
    • Callbacks
    • Promises
    • Coroutines
    

    View Slide

28. Event Loop
    

    View Slide

29. Launch Initialise
    Running
    Cleanup Terminate
    Register
    callbacks
    Receive
    events
    

    View Slide

30. import asyncio
    # Init code goes here.
    loop = asyncio.get_event_loop()
    loop.run_forever()
    

    View Slide

31. import asyncio
    # Init code goes here.
    def print_and_exit():
    print('Hello asyncio!')
    asyncio.get_event_loop().stop()
    loop = asyncio.get_event_loop()
    loop.call_soon(print_and_exit)
    loop.run_forever()
    

    View Slide

32. CALLBACK
    When the going gets rough.
    

    View Slide

33. Pyramid of Doom
    $(function () {
    $("button").click(function (e) {
    $.get(URL, function (data) {
    $(".list").each(function () {
    $(this).click(function () {
    setTimeout(function () {
    alert("Top of the world");
    }, 1000);
    });
    });
    });
    });
    });
    

    View Slide

34. WELCOME TO
    CALLBACK HELL
    

    View Slide

35. def send_repos_request():
    ...
    task = ...
    task.add_done_callback(...)
    def read_repos(future):
    ... = future.result()
    ...
    task = ...
    task.add_done_callback(...)
    def send_forked_repo_requests(future):
    ... = future.result()
    ...
    task = ...
    task.add_done_callback(...)
    def read_forked_repos(future):
    ... = future.result()
    ...
    task = ...
    task.add_done_callback(...)
    

    View Slide

36. Promise
    

    View Slide

37. def main():
    (send_repos_request()
    .then(read_repos)
    .then(send_forked_repo_requests, cleanup)
    .then(read_forked_repo_requests)
    .then(print_source_names)
    .done(cleanup))
    

    View Slide

38. def main():
    (send_repos_request()
    .then(read_repos)
    .then(send_forked_repo_requests, cleanup)
    .then(read_forked_repo_requests)
    .then(print_source_names)
    .done(cleanup))
    

    View Slide

39. def main():
    (send_repos_request()
    .then(read_repos)
    .then(send_forked_repo_requests, cleanup)
    .then(read_forked_repo_requests)
    .then(print_source_names)
    .done(cleanup))
    

    View Slide

40. def main():
    (send_repos_request()
    .then(read_repos)
    .then(send_forked_repo_requests, cleanup)
    .then(read_forked_repo_requests)
    .then(print_source_names)
    .done(cleanup))
    

    View Slide

41. def main():
    (send_repos_request()
    .then(read_repos)
    .catch(cleanup)
    .then(send_forked_repo_requests)
    .then(read_forked_repo_requests)
    .then(print_source_names)
    .done(cleanup))
    

    View Slide

42. def main():
    (send_repos_request()
    .then(read_repos)
    .catch(cleanup)
    .then(send_forked_repo_requests)
    .then(read_forked_repo_requests)
    .then(print_source_names)
    .done(cleanup))
    

    View Slide

43. ???????
    

    View Slide

44. Coroutine
    

    View Slide

45. async def run(session):
    response = await send_repos_request(session)
    try:
    data = await read_repos(response)
    except APIError:
    return
    responses = await send_forked_repo_requests(session, data)
    datasets = await read_forked_repo_requests(responses)
    print_source_names(datasets)
    

    View Slide

46. async def run(session):
    response = await send_repos_request(session)
    try:
    data = await read_repos(response)
    except APIError:
    return
    responses = await send_forked_repo_requests(session, data)
    datasets = await read_forked_repo_requests(responses)
    print_source_names(datasets)
    

    View Slide

47. View Slide

48. Everything Has a Counterpart
    • async def / await
    • async with
    • async for (Python 3.6)
    

    View Slide

49. class AsyncContextManager:
    async def __aenter__(self):
    await log('entering context')
    async def __aexit__(self, exc_type, exc, tb):
    await log('exiting context')
    

    View Slide

50. class AsyncIterable:
    def __aiter__(self):
    return self
    async def __anext__(self):
    data = await ...
    if data:
    return data
    else:
    raise StopAsyncIteration
    

    View Slide

51. Python
    asyncio.
    

    View Slide

52. Wrapping Up
    • What is asynchrony
    • How asynchrony works
    • Asynchrony and asyncio
    

    View Slide

53. Further Watching
    • A gentle introduction to asyncio
    • What in the World is Asyncio?
    • Topics of Interest (Python Asyncio)
    • You Might Not Want Async
    • and more…!
    

    View Slide

54. View Slide

55. View Slide