Tweet
Tweet

Slide 1

Slide 1 text

You Might Not Want Async

Slide 2

Slide 2 text

Quick Questions • Concurrency with Python • Threads • Multi-processing • Single-thread asynchrony • asyncio

Slide 3

Slide 3 text

ࣗݾ঺հ (PyCon JP Ver.) • ৉ ࢠሯ • ͫΐ͏ ͪʔͽΜ • @uranusjr

Slide 4

Slide 4 text

Me • Call me TP • Follow @uranusjr • https://uranusjr.com

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

http://macdown.uranusjr.com

Slide 8

Slide 8 text

www. .com

Slide 9

Slide 9 text

10–11 June 2017 (Ծ) 7

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

meh

Slide 13

Slide 13 text

2014 Python 3.4 OSDC.tw PyCon APAC March April May

Slide 14

Slide 14 text

2016

Slide 15

Slide 15 text

Got me thinking

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Synchronous

Slide 18

Slide 18 text

(Single-Threaded) Async

Slide 19

Slide 19 text

Sync vs. Async

Slide 20

Slide 20 text

Absolute magic

Slide 21

Slide 21 text

Before After

Slide 22

Slide 22 text

Module 5 - Doctor Faustus by Christopher Marlowe

Slide 23

Slide 23 text

Not For You • Infects the whole program • Async is not parallelism • Third party support

Slide 24

Slide 24 text

import sqlite3 def read_data(dbname): con = sqlite3.connect(dbname) cur = con.cursor() cur.execute('SELECT * FROM data OFFSET 0 LIMIT 1') data = cur.fetchone() cur.close() con.close() return data

Slide 25

Slide 25 text

import aioodbc async def read_data(dbname): con = await aioodbc.connect( dsn='Driver=SQLite;Database={}'.format(dbname), ) cur = await con.cursor() await cur.execute('SELECT * FROM data OFFSET 0 LIMIT 1') data = await cur.fetchone() await cur.close() await con.close() return data

Slide 26

Slide 26 text

from .db import read_data def main(): data = read_data('data.sqlite3') print(data) main()

Slide 27

Slide 27 text

import asyncio from .db import read_data async def main(): data = await read_data('db.sqlite3') print(data) loop = asyncio.get_event_loop() loop.run_until_complete(main()) loop.close()

Slide 28

Slide 28 text

U+1F937 SHRUG (Unicode 9.0)

Slide 29

Slide 29 text

RELAX PEOPLE I’M JUST GETTING STARTED

Slide 30

Slide 30 text

import asyncio from .db import read_data async def main(): data = read_data('db.sqlite3') print(data) loop = asyncio.get_event_loop() loop.run_until_complete(main()) loop.close()

Slide 31

Slide 31 text

(demo) $ python demo.py

Slide 32

Slide 32 text

import asyncio from .db import read_data async def main(): data = read_data('db.sqlite3') print(data) loop = asyncio.get_event_loop() loop.run_until_complete(main()) loop.close()

Slide 33

Slide 33 text

import asyncio from .db import read_data async def main(): data = await read_data('db.sqlite3') print(data) loop = asyncio.get_event_loop() loop.run_until_complete(main()) loop.close()

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

I KNOW WRITE UNIT TESTS

Slide 36

Slide 36 text

GOOD LUCK WITH THAT

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

import asyncio import time async def do_something(): print('Before', time.monotonic()) await asyncio.sleep(2) print('After ', time.monotonic())  await do_something() Before 199208.190632853 After 199210.192092425

Slide 40

Slide 40 text

import unittest class MyTestCase(unittest.TestCase): async def test_do_something(self): await do_something() if __name__ == '__main__': unittest.main()

Slide 41

Slide 41 text

(demo) $ python tests.py . ---------------------------------------------------- Ran 1 test in 0.002s OK

Slide 42

Slide 42 text

(demo) $ python tests.py . ---------------------------------------------------- Ran 1 test in 0.002s OK

Slide 43

Slide 43 text

What Happened? • unittest does not know about asyncio • Coroutine methods are executed “normally” • Called, but not executed (awaited)

Slide 44

Slide 44 text

WHAT IF I TOLD YOU THIS IS GONNA BE TOUGH

Slide 45

Slide 45 text

import asyncio import functools def asynchronous(func): @functools.wraps(func) def asynchronous_inner(*args, **kwargs): loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) try: loop.run_until_complete(func(*args, **kwargs)) finally: loop.close() return asynchronous_inner

Slide 46

Slide 46 text

import unittest class MyTestCase(unittest.TestCase): @asynchronous async def test_do_something(self): await do_something() if __name__ == '__main__': unittest.main()

Slide 47

Slide 47 text

(demo) $ python tests.py Before 51728.420457105 After 51730.424515145 . ---------------------------------------------------- Ran 1 test in 2.006s OK

Slide 48

Slide 48 text

Tips • Beware of warning output • Especially if you redirect • Add coverage report to testing code • Consider pip install asynctest

Slide 49

Slide 49 text

Or use pytest instead

Slide 50

Slide 50 text

import time async def test_do_something(): before = time.monotonic() await do_something() delta_t = time.monotonic() - before assert -0.01 < delta_t < 0.01

Slide 51

Slide 51 text

(demo) $ py.test tests.py ====================== test session starts ====================== platform darwin -- Python 3.5.1, pytest-2.9.1, py-1.4.31, pluggy-0.3.1 collected 0 items / 1 errors ============================ ERRORS ============================= ___________________ ERROR collecting tests.py____________________ > for i, x in enumerate(self.obj()): E TypeError: 'coroutine' object is not iterable python3.5/site-packages/_pytest/python.py:765: TypeError ==================== 1 error in 0.15 seconds ====================

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

import time import pytest @pytest.mark.asyncio async def test_do_something(capsys): before = time.monotonic() await do_something() delta_t = time.monotonic() - before assert -0.01 < delta_t < 0.01

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

asyncio • Boilerplate • Error-prone • Immature API (I think)

Slide 58

Slide 58 text

import requests def collect_contents(urls): contents = [] for url in urls: resp = requests.get(url) if resp.status_code != 200: continue content = resp.text contents.append(content) return contents

Slide 59

Slide 59 text

import aiohttp async def collect_contents(urls): contents = [] with aiohttp.ClientSession() as session: for url in urls: async with session.get(url) as rest: if resp.status != 200: continue content = await resp.text() contents.append(content) return contents

Slide 60

Slide 60 text

㖶װ׌ַ׵ׁ

Slide 61

Slide 61 text

import aiohttp async def collect_contents(urls): contents = [] with aiohttp.ClientSession() as session: for url in urls: async with session.get(url) as rest: if resp.status != 200: continue content = await resp.text() contents.append(content) return contents

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

import asyncio import aiohttp async def collect_contents(urls): coroutines = [] with aiohttp.ClientSession() as session: for url in urls: async with session.get(url) as rest: if resp.status != 200: continue coroutines.append(resp.text()) contents = await asyncio.gather(*coroutines) return contents

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

No content

Slide 67

Slide 67 text

LET’S TRY SOMETHING “DIFFERENT”

Slide 68

Slide 68 text

Alternatives • concurrent & multiprocessing • Greenlets • Similar idea, but less infectious • C extension • threading • Standard I/O

Slide 69

Slide 69 text

http://greenlet.readthedocs.io

Slide 70

Slide 70 text

GEVENT MEH

Slide 71

Slide 71 text

ROUTINES ROUTINES EVERYWHERE

Slide 72

Slide 72 text

package main import ("fmt"; "time") func doSomething() { time.Sleep(2 * time.Second) fmt.Println(time.Now(), "Slept") } func main() { doSomething() fmt.Println(time.Now(), "OK") } 00:00:00 Slept 00:00:00 OK

Slide 73

Slide 73 text

package main import ("fmt"; "time") func doSomething() { time.Sleep(2 * time.Second) fmt.Println(time.Now(), "Slept") } func main() { go doSomething() fmt.Println(time.Now(), "OK") } 00:00:00 OK

Slide 74

Slide 74 text

package main import ("fmt"; "time") var sem = make(chan bool) func doSomething() { time.Sleep(2 * time.Second) fmt.Println(time.Now(), "Slept") sem  true } func main() { go doSomething() fmt.Println(time.Now(), "OK") sem } 00:00:00 OK 00:00:02 Slept

Slide 75

Slide 75 text

The Go Model • Boilerplate • Error-prone • Immature API (I think)

Slide 76

Slide 76 text

"""Do something. Synchronous version.""" import datetime import time def do_something(): time.sleep(2) print(datetime.datetime.now(), 'Slept') do_something() print(datetime.datetime.now(), 'Done')

Slide 77

Slide 77 text

"""Do something. Asynchronous version.""" import asyncio import datetime async def do_something(): await asyncio.sleep(2) print(datetime.datetime.now(), 'Slept') loop = asyncio.get_event_loop() task = loop.create_task(do_something()) print(datetime.datetime.now(), 'Done') loop.run_until_complete(asyncio.wait([task])) loop.close()

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

"""Do something. Synchronous version.""" import datetime import time def do_something(): time.sleep(2) print(datetime.datetime.now(), 'Slept') do_something() print(datetime.datetime.now(), 'Done')

Slide 80

Slide 80 text

"""What if I can just write this?""" import asyncio import datetime import time async def do_something(): await time.sleep(2) print(datetime.datetime.now(), 'Slept') await do_something() print(datetime.datetime.now(), 'Done') asyncio.run_event_loop()

Slide 81

Slide 81 text

I know, it’s not really possible.

Slide 82

Slide 82 text

At least we can dream. Or wait until Python 6.0?

Slide 83

Slide 83 text

Recap • Rant • Moar rant • Susceptible advice • Unrealistic dream

Slide 84

Slide 84 text

But Seriously • Asynchrony is not the silver bullet • It makes you jump through loops • There are alternatives • Fingers crossed

Slide 85

Slide 85 text

No content