Upgrade to Pro — share decks privately, control downloads, hide ads and more …

如何自幹一個 Python Coroutine

Jason
June 13, 2018
170

如何自幹一個 Python Coroutine

要達成非同步(Asynchronous) I/O 有很多種策略,經常聽到的是使用多執行緒(multithreading)達到非同步。雖然 GIL(Global Interpreter Lock) 讓 Python multithreading 更適合 I/O 頻繁的應用(concurrency),但實際上頻繁的上下文切換(context-switch)卻消耗了更多時間。

另一種策略是基於協同程序(Coroutine)來實現非同步,在 single-thread 下允許程式來決定程式執行的順序,因此可以更有效的利用 CPU 處理時間。

本次簡短分享將會示範如何建立一個 Asynchronous I/O (non-blocking, callback, event-loop) 接著實做一個 Coroutine 來改寫原本 callback 方式。

https://www.meetup.com/Kaohsiung-Python-Meetup/events/251634923/

Jason

June 13, 2018
Tweet

Transcript

  1. 故事是這樣的 ಭᑤ PyConTW 牫 Ꮈ绗 Channels 牫牫 覍ݶྍ 牫牫牫 Coroutine

    牫牫牫牫 樄ত፡ԧӞञ୽粙牧簁盅磪讨睞眖牦
  2. Asynchronous • ᴥलୗग़ᤈ纷 (ྯ㮆藶穩樄Ӟ㮆 process, ֕᩼㬵᩼ग़ cpu context-switch 疰᩼ṛ) •

    ᴥलୗԪग़ᤈ纷ग़䁆ᤈ姼 (樄ग़㮆ᤈ纷膏䁆ᤈ姼牧蝨౮ dead lock, race condition, 疪ٌ蒂ቘوݶ䩚ᥜ碻蝡犚㺔氂疰䨝ๅ瑥᯿牧犖犋অ䌃) • 覍ᴥलୗԪկ詴㵕 (㻌Ӟ蝅瑹䲒礚磪篷翕᪠ IO Ԫկ咳ኞ牧፜݄ԧ context-switch, process copy 缛牧砰䌃蕦褾牧ܻࢩฎӞկԪ眐ᤩ藉咳牧磪Ԫ眐䷱螭蒂ቘ疰ض懿 袅牧ӥ稞藉咳ٚ㬵礬硁ض獮制眲㬵蒂ቘ) ex: Twisted • 覍ᴥलୗ Coroutine (猟ฎᴥलୗӞ䰬ፗ薪牧䟖㻌Ӟ蝅瑹䲒礚Ԫկ牧֕ےӤ Coroutine) ex: gevent, asyncio
  3. Asynchronous • ᴥलୗग़ᤈ纷 (ྯ㮆藶穩樄Ӟ㮆 process, ֕᩼㬵᩼ग़ cpu context-switch 疰᩼ṛ) •

    ᴥलୗԪग़ᤈ纷ग़䁆ᤈ姼 (樄ग़㮆ᤈ纷膏䁆ᤈ姼牧蝨౮ dead lock, race condition, 疪ٌ蒂ቘوݶ䩚ᥜ碻蝡犚㺔氂疰䨝ๅ瑥᯿牧犖犋অ䌃) • 覍ᴥलୗԪկ詴㵕 (㻌Ӟ蝅瑹䲒礚磪篷翕᪠ IO Ԫկ咳ኞ牧፜݄ԧ context-switch, process copy 缛牧砰䌃蕦褾牧ܻࢩฎӞկԪ眐ᤩ藉咳牧磪Ԫ眐䷱螭蒂ቘ疰ض懿 袅牧ӥ稞藉咳ٚ㬵礬硁ض獮制眲㬵蒂ቘ) ex: Twisted • 覍ᴥलୗ Coroutine (猟ฎᴥलୗӞ䰬ፗ薪牧䟖㻌Ӟ蝅瑹䲒礚Ԫկ牧֕ےӤ Coroutine) ex: gevent, asyncio
  4. # sync def get(path): s = socket.socket() s.connect(('localhost', 5000)) request

    = 'GET %s HTTP/1.0\r\n\r\n' % path s.send(request.encode()) chunks = [] while True: chunk = s.recv(1000) if chunk: chunks.append(chunk) else: body = (b''.join(chunks)).decode() print(body.split('\n')[0]) return
  5. Multithreading with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor: future_to_url = {executor.submit(get, url): url

    for url in URLS} for future in concurrent.futures.as_completed(future_to_url): url = future_to_url[future] try: data = future.result() except Exception as exc: print('%r generated an exception: %s' % (url, exc)) print('multithreading took %.1f sec' % (time.time() - start))
  6. from selectors import DefaultSelector, EVENT_WRITE, EVENT_READ selector = DefaultSelector() def

    get_non_blocking(path): s = socket.socket() s.setblocking(False) try: s.connect(('localhost', 5000)) except BlockingIOError: pass Non-blocking
  7. def get_callback(path): ... callback = lambda: connected(s, path) ... callback()

    def connected(s, path): ... callback = lambda: readable(s, chunks) ... callback() def readable(s, chunks): ... if chunk: ... callback = lambda: readable(s, chunks) ... callback() else: ... return
  8. Event-loop Javascript’s Event-loop sample ֦狶অ盅ݞ౯ (by TP) A 咳ኞ, 䁆ᤈ

    B https://pjchender.blogspot.com/2017/08/javascript-learn-event-loop-stack-queue.html
  9. n_jobs = 0 def get_eventloop(path): global n_jobs n_jobs += 1

    ... callback = lambda: connected_event(s, path) # closure selector.register(s.fileno(), EVENT_WRITE, data=callback) def connected_event(s, path): ... callback = lambda: readable_event(s, chunks) def readable_event(s, chunks): global n_jobs ... if chunk: ... else: n_jobs -= 1
  10. selector = DefaultSelector() c_n_jobs = 0 class Future: def __init__(self):

    self.callbacks = None def resolve(self): self.callbacks() def __await__(self): yield self
  11. class Task: def __init__(self, coro): self.coro = coro self.step() def

    step(self): try: f = self.coro.send(None) except StopIteration: return f.callbacks = self.step
  12. async def get_coroutines(path): global n_jobs n_jobs += 1 ... f

    = Future() selector.register(s.fileno(), EVENT_WRITE, data=f) await f selector.unregister(s.fileno()) request = 'GET %s HTTP/1.0\r\n\r\n' % path s.send(request.encode()) chunks = [] while True: f = Future() selector.register(s.fileno(), EVENT_READ, data=f) await f selector.unregister(s.fileno()) chunk = s.recv(1000) if chunk: chunks.append(chunk) else: break body = (b''.join(chunks)).decode() print(body.split('\n')[0]) n_jobs -= 1