Slide 1
Slide 1 text
You Might Not Want Async
Slide 2
Slide 2 text
Quick Questions
• Concurrency with Python
• Threads
• Multi-processing
• Single-thread asynchrony
• asyncio
Slide 3
Slide 3 text
ࣗݾհ (PyCon JP Ver.)
• ࢠሯ
• ͫΐ͏ ͪʔͽΜ
• @uranusjr
Slide 4
Slide 4 text
Me
• Call me TP
• Follow @uranusjr
• https://uranusjr.com
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
No content
Slide 7
Slide 7 text
http://macdown.uranusjr.com
Slide 8
Slide 8 text
www. .com
Slide 9
Slide 9 text
10–11 June 2017 (Ծ)
7
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
meh
Slide 13
Slide 13 text
2014
Python 3.4 OSDC.tw PyCon APAC
March April May
Slide 14
Slide 14 text
2016
Slide 15
Slide 15 text
Got me
thinking
Slide 16
Slide 16 text
No content
Slide 17
Slide 17 text
Synchronous
Slide 18
Slide 18 text
(Single-Threaded) Async
Slide 19
Slide 19 text
Sync vs. Async
Slide 20
Slide 20 text
Absolute
magic
Slide 21
Slide 21 text
Before After
Slide 22
Slide 22 text
Module 5 - Doctor Faustus by Christopher Marlowe
Slide 23
Slide 23 text
Not For You
• Infects the whole program
• Async is not parallelism
• Third party support
Slide 24
Slide 24 text
import sqlite3
def read_data(dbname):
con = sqlite3.connect(dbname)
cur = con.cursor()
cur.execute('SELECT * FROM data OFFSET 0 LIMIT 1')
data = cur.fetchone()
cur.close()
con.close()
return data
Slide 25
Slide 25 text
import aioodbc
async def read_data(dbname):
con = await aioodbc.connect(
dsn='Driver=SQLite;Database={}'.format(dbname),
)
cur = await con.cursor()
await cur.execute('SELECT * FROM data OFFSET 0 LIMIT 1')
data = await cur.fetchone()
await cur.close()
await con.close()
return data
Slide 26
Slide 26 text
from .db import read_data
def main():
data = read_data('data.sqlite3')
print(data)
main()
Slide 27
Slide 27 text
import asyncio
from .db import read_data
async def main():
data = await read_data('db.sqlite3')
print(data)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
Slide 28
Slide 28 text
U+1F937 SHRUG (Unicode 9.0)
Slide 29
Slide 29 text
RELAX PEOPLE
I’M JUST GETTING STARTED
Slide 30
Slide 30 text
import asyncio
from .db import read_data
async def main():
data = read_data('db.sqlite3')
print(data)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
Slide 31
Slide 31 text
(demo) $ python demo.py
Slide 32
Slide 32 text
import asyncio
from .db import read_data
async def main():
data = read_data('db.sqlite3')
print(data)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
Slide 33
Slide 33 text
import asyncio
from .db import read_data
async def main():
data = await read_data('db.sqlite3')
print(data)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
Slide 34
Slide 34 text
No content
Slide 35
Slide 35 text
I KNOW
WRITE UNIT TESTS
Slide 36
Slide 36 text
GOOD LUCK
WITH THAT
Slide 37
Slide 37 text
No content
Slide 38
Slide 38 text
No content
Slide 39
Slide 39 text
import asyncio
import time
async def do_something():
print('Before', time.monotonic())
await asyncio.sleep(2)
print('After ', time.monotonic())
await do_something()
Before 199208.190632853
After 199210.192092425
Slide 40
Slide 40 text
import unittest
class MyTestCase(unittest.TestCase):
async def test_do_something(self):
await do_something()
if __name__ == '__main__':
unittest.main()
Slide 41
Slide 41 text
(demo) $ python tests.py
.
----------------------------------------------------
Ran 1 test in 0.002s
OK
Slide 42
Slide 42 text
(demo) $ python tests.py
.
----------------------------------------------------
Ran 1 test in 0.002s
OK
Slide 43
Slide 43 text
What Happened?
• unittest does not know about asyncio
• Coroutine methods are executed “normally”
• Called, but not executed (awaited)
Slide 44
Slide 44 text
WHAT IF I TOLD YOU
THIS IS GONNA BE TOUGH
Slide 45
Slide 45 text
import asyncio
import functools
def asynchronous(func):
@functools.wraps(func)
def asynchronous_inner(*args, **kwargs):
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
loop.run_until_complete(func(*args, **kwargs))
finally:
loop.close()
return asynchronous_inner
Slide 46
Slide 46 text
import unittest
class MyTestCase(unittest.TestCase):
@asynchronous
async def test_do_something(self):
await do_something()
if __name__ == '__main__':
unittest.main()
Slide 47
Slide 47 text
(demo) $ python tests.py
Before 51728.420457105
After 51730.424515145
.
----------------------------------------------------
Ran 1 test in 2.006s
OK
Slide 48
Slide 48 text
Tips
• Beware of warning output
• Especially if you redirect
• Add coverage report to testing code
• Consider pip install asynctest
Slide 49
Slide 49 text
Or use pytest instead
Slide 50
Slide 50 text
import time
async def test_do_something():
before = time.monotonic()
await do_something()
delta_t = time.monotonic() - before
assert -0.01 < delta_t < 0.01
Slide 51
Slide 51 text
(demo) $ py.test tests.py
====================== test session starts ======================
platform darwin -- Python 3.5.1, pytest-2.9.1, py-1.4.31,
pluggy-0.3.1
collected 0 items / 1 errors
============================ ERRORS =============================
___________________ ERROR collecting tests.py____________________
> for i, x in enumerate(self.obj()):
E TypeError: 'coroutine' object is not iterable
python3.5/site-packages/_pytest/python.py:765: TypeError
==================== 1 error in 0.15 seconds ====================
Slide 52
Slide 52 text
No content
Slide 53
Slide 53 text
import time
import pytest
@pytest.mark.asyncio
async def test_do_something(capsys):
before = time.monotonic()
await do_something()
delta_t = time.monotonic() - before
assert -0.01 < delta_t < 0.01
Slide 54
Slide 54 text
No content
Slide 55
Slide 55 text
No content
Slide 56
Slide 56 text
No content
Slide 57
Slide 57 text
asyncio
• Boilerplate
• Error-prone
• Immature API (I think)
Slide 58
Slide 58 text
import requests
def collect_contents(urls):
contents = []
for url in urls:
resp = requests.get(url)
if resp.status_code != 200:
continue
content = resp.text
contents.append(content)
return contents
Slide 59
Slide 59 text
import aiohttp
async def collect_contents(urls):
contents = []
with aiohttp.ClientSession() as session:
for url in urls:
async with session.get(url) as rest:
if resp.status != 200:
continue
content = await resp.text()
contents.append(content)
return contents
Slide 60
Slide 60 text
㖶װַׁ
Slide 61
Slide 61 text
import aiohttp
async def collect_contents(urls):
contents = []
with aiohttp.ClientSession() as session:
for url in urls:
async with session.get(url) as rest:
if resp.status != 200:
continue
content = await resp.text()
contents.append(content)
return contents
Slide 62
Slide 62 text
No content
Slide 63
Slide 63 text
import asyncio
import aiohttp
async def collect_contents(urls):
coroutines = []
with aiohttp.ClientSession() as session:
for url in urls:
async with session.get(url) as rest:
if resp.status != 200:
continue
coroutines.append(resp.text())
contents = await asyncio.gather(*coroutines)
return contents
Slide 64
Slide 64 text
No content
Slide 65
Slide 65 text
No content
Slide 66
Slide 66 text
No content
Slide 67
Slide 67 text
LET’S TRY SOMETHING
“DIFFERENT”
Slide 68
Slide 68 text
Alternatives
• concurrent & multiprocessing
• Greenlets
• Similar idea, but less infectious
• C extension
• threading
• Standard I/O
Slide 69
Slide 69 text
http://greenlet.readthedocs.io
Slide 70
Slide 70 text
GEVENT
MEH
Slide 71
Slide 71 text
ROUTINES
ROUTINES EVERYWHERE
Slide 72
Slide 72 text
package main
import ("fmt"; "time")
func doSomething() {
time.Sleep(2 * time.Second)
fmt.Println(time.Now(), "Slept")
}
func main() {
doSomething()
fmt.Println(time.Now(), "OK")
}
00:00:00 Slept
00:00:00 OK
Slide 73
Slide 73 text
package main
import ("fmt"; "time")
func doSomething() {
time.Sleep(2 * time.Second)
fmt.Println(time.Now(), "Slept")
}
func main() {
go doSomething()
fmt.Println(time.Now(), "OK")
}
00:00:00 OK
Slide 74
Slide 74 text
package main
import ("fmt"; "time")
var sem = make(chan bool)
func doSomething() {
time.Sleep(2 * time.Second)
fmt.Println(time.Now(), "Slept")
sem true
}
func main() {
go doSomething()
fmt.Println(time.Now(), "OK")
sem
}
00:00:00 OK
00:00:02 Slept
Slide 75
Slide 75 text
The Go Model
• Boilerplate
• Error-prone
• Immature API (I think)
Slide 76
Slide 76 text
"""Do something. Synchronous version."""
import datetime
import time
def do_something():
time.sleep(2)
print(datetime.datetime.now(), 'Slept')
do_something()
print(datetime.datetime.now(), 'Done')
Slide 77
Slide 77 text
"""Do something. Asynchronous version."""
import asyncio
import datetime
async def do_something():
await asyncio.sleep(2)
print(datetime.datetime.now(), 'Slept')
loop = asyncio.get_event_loop()
task = loop.create_task(do_something())
print(datetime.datetime.now(), 'Done')
loop.run_until_complete(asyncio.wait([task]))
loop.close()
Slide 78
Slide 78 text
No content
Slide 79
Slide 79 text
"""Do something. Synchronous version."""
import datetime
import time
def do_something():
time.sleep(2)
print(datetime.datetime.now(), 'Slept')
do_something()
print(datetime.datetime.now(), 'Done')
Slide 80
Slide 80 text
"""What if I can just write this?"""
import asyncio
import datetime
import time
async def do_something():
await time.sleep(2)
print(datetime.datetime.now(), 'Slept')
await do_something()
print(datetime.datetime.now(), 'Done')
asyncio.run_event_loop()
Slide 81
Slide 81 text
I know, it’s not really
possible.
Slide 82
Slide 82 text
At least we can dream.
Or wait until Python 6.0?
Slide 83
Slide 83 text
Recap
• Rant
• Moar rant
• Susceptible advice
• Unrealistic dream
Slide 84
Slide 84 text
But Seriously
• Asynchrony is not the silver bullet
• It makes you jump through loops
• There are alternatives
• Fingers crossed
Slide 85
Slide 85 text
No content