asyncio today & tomorrow

Slide 1

Slide 1 text

asyncio today & tomorrow EuroPython 2018

Slide 2

Slide 2 text

Hi, I'm Yury • Python Core Developer since 2013 • PEPs 362, 492, 525, 530, 550, 567 • asyncio maintainer • uvloop, asyncpg • EdgeDB • Twitter, GitHub: @1st1

Slide 3

Slide 3 text

brief asyncio history Part I

Slide 4

Slide 4 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB • Python 3.3 ~2013   Guido works on Tulip • Tulip is a reference implementation  of PEP 3156 • Inspired by Twisted & co • Becomes part of Python 3.4 Brief asyncio History Python 3.3

Slide 5

Slide 5 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB • Provisional • Low-level APIs: Protocols, Transports,   Futures, and callbacks • Coroutines via yield from • High-level APIs: Streams,   Subprocesses, etc Brief asyncio History Python 3.4

Slide 6

Slide 6 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB Brief asyncio History Python 3.5 • async / await • asyncio is still provisional • A bunch of new APIs • uvloop • New framework: Curio  hm, what can we learn from it?

Slide 7

Slide 7 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB Brief asyncio History Python 3.6 • No longer provisional :( • async generators • We've ﬁxed get_event_loop() • New low-level APIs... • New framework: Trio  hm, what can we learn from it?

Slide 8

Slide 8 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB Brief asyncio History Python 3.7 • Context Variables—contextvars • asyncio's own code uses async/await • asyncio.run() [thanks, Curio!] • Grab bag: sendﬁle, start TLS, create_task(),  get_running_loop(), BuﬀeredProtocol, etc • Better third-party event loops support

Slide 9

Slide 9 text

async / await Part II

Slide 10

Slide 10 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB async / await asyncio layers normal asyncio.run()  asyncio.gather()  asyncio.create_task() asyncio.sleep()  streams API  hardcore loop.*()  protocols & transports  asyncio.Future

Slide 11

Slide 11 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB async / await asyncio.run()

Slide 12

Slide 12 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB asyncio.run() async / await

Slide 13

Slide 13 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB asyncio.run() async / await

Slide 14

Slide 14 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB async / await use it! • with asyncio.run() you don't   need the loop • have just one entry point • use async / await for everything • don't pass a reference to the loop  anywhere

Slide 15

Slide 15 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB async / await serve_forever()

Slide 16

Slide 16 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB async / await serve_forever()

Slide 17

Slide 17 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB async / await get_running_loop()

Slide 18

Slide 18 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB async / await get_running_loop()

Slide 19

Slide 19 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB async / await don'ts • don't use @coroutine,   we will remove it soon-ish • don't use low-level APIs (futures,  call_soon(), call_later(), transports, protocols, event loop)  unless you have to.

Slide 20

Slide 20 text

good async / await code Part III

Slide 21

Slide 21 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB good code What code is good? • That you can write quick? • Maintainable? • Beautiful? • Robust? • Fast? (subjective) (subjective) (subjective) 2 3 3

Slide 22

Slide 22 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB good code Let's talk about fast & robust monitoring production in

Slide 23

Slide 23 text

good code Is async code good? ... ... ... ... time

Slide 24

Slide 24 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB good code contextvars • PEP 550, 4 diﬀerent revisions • PEP 567 • ~900 emails on python-ideas  and python-dev

Slide 25

Slide 25 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB good code contextvars • warning: it's magic • shipped with Python 3.7 • standard library module • full asyncio support • decimal context uses it

Slide 26

Slide 26 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB good code contextvars

Slide 27

Slide 27 text

... ... ... ... time good code contextvars ID: 42 ID: 42 ID: 42 ID: 42 ID: 42 ID: 17 ID: 92 ID: 63 ID: 18 ID: 17 ID: 63 ID: 18 ID: 87 ID: 34 ID: 90 ID: 54 ID: 78 ID: 22 ID: 11 ID: 56 ID: 65 ID: 45 ID: 23 ID: 17 ID: 92 ID: 63 ID: 18 ID: 78 ID: 11 ID: 56 ID: 65 ID: 45 ID: 22 ID: 43 ID: 34 ID: 90 ID: 90 ID: 12 ID: 18 ID: 23 ID: 38 ID: 49 ID: 02 ID: 99

Slide 28

Slide 28 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB good code Use contextvars for • monitoring:  e.g. how long some operations take • localization:   e.g. current language for HTTP request • security:  e.g. current user or permissions • debug: • execution context:   e.g. decimal context & numpy error context

Slide 29

Slide 29 text

what's next Part IV

Slide 30

Slide 30 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB what's next Let's talk about Trio • new library by Nathaniel J. Smith • designed from scratch • incompatible with asyncio • hard focus on usability • got many things right! • youtu.be/oLkfnc_UMcE

Slide 31

Slide 31 text

what's next back in 1958...

Slide 32

Slide 32 text

what's next 10 years later, 1968 Dijkstra:   "Go To Statement  Considered Harmful" Structured  Programming

Slide 33

Slide 33 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB what's next Back to Trio: nurseries

Slide 34

Slide 34 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB what's next Trio's nurseries are cool! • almost no out of order execution • control ﬂow is traceable • exceptions are never lost • with and try blocks work • they solve the "goto problem"  in concurrency

Slide 35

Slide 35 text

what's next Back to asyncio ... ... ... ... time

Slide 36

Slide 36 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB what's next a typical asyncio library ... HTTP client

Slide 37

Slide 37 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB what's next a typical asyncio library • most libraries  do nothing • some invent  ad-hoc half-working  solutions • this is a bug magnet  in asyncio

Slide 38

Slide 38 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB ... HTTP client what's next what we need

Slide 39

Slide 39 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB what's next an idea for Python 3.8 • new low-level API for libraries  and frameworks • something I'm thinking about  to land in Python 3.8 loop.create_supervisor()

Slide 40

Slide 40 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB • returns an asynchronous context  manager • supervisor mirrors all asyncio event  loop APIs • can be passed from one coroutine  to another what's next create_supervisor()

Slide 41

Slide 41 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB what's next create_supervisor()

Slide 42

Slide 42 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB what's next create_supervisor() • any unhandled exception in a  callback or transport (IO) cleans  up all resources allocated through  the supervisor • easier to implement cancellation   and cleanup logic on top • third-party loops are in control

Slide 43

Slide 43 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB what's next another idea for Python 3.8 asyncio.TaskGroup() [thanks, Curio!]

Slide 44

Slide 44 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB what's next TaskGroup() • will use the new  loop.create_supervisor()   under the hood • more convenient API than   asyncio.gather() • easy to schedule tasks in "buckets"

Slide 45

Slide 45 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB what's next So what to expect from 3.8? 1. we hear you! 2. documentation improvements 3. likely: loop.create_supervisor() 4. likely: asyncio.TaskGroup() 5. maybe: event loop low-level  tracing API

Slide 46

Slide 46 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB what's next So what to expect from 3.8? 6. maybe: timeout and cancel scopes  like in Trio 7. maybe: new streams API 8. maybe: add a context manager for shield() 9. likely: SSL over SSL 10.Make CancelledError a BaseException

Slide 47

Slide 47 text

Yury Selivanov @1st1 EuroPython 2018 EdgeDB Questions? Thank you!