[PyCon KR 2017] Meet aiotools: asyncio idiom library

Describes the aiotools library – its background, key features, and code examples about how to use it. Also introduces other helper/alternative asyncio libraries.

Transcript

1. Meet aiotools: asyncio idiom library 김준기 CTO, Lablup Inc. / 글로벌 오픈프론티어 4기
2. github.com/achimnol
3. asyncio § 이제 다 아시죠? § 하지만 실제로 쓰다보면... § Python 3.6부터 가능해진 것 — async generator async def fetch_data(): async with connect_db() as conn: async for row in conn.fetch(query): yield row async for row in fetch_data(): print(row)
4. aiotools! § Lablup.AI 플랫폼 개발하면서 반복되는 코드를 리팩토링한 결과물 § Python 3.6의 async generator를 잘 써보자! § aiohttp, aiodocker 프로젝트에서 배운 CI 기법들을 적용해보자! § (asyncio 프로젝트들이 핫하니 좋은 이름 빨리 선점해볼까?!) ⭐는 사랑입니다! https://github.com/achimnol/aiotools
5. Reminder: context manager class MyContext: def __init__(self, ...): self.resource = Resource(...) def __enter__(self): self.resource.acquire() return self.resource def __exit__(self, exc_type, exc_val, tb): self.resource.release() @contextlib.contextmanager def my_context(...): resource = Resource(...) resource.acquire() try: yield resource finally: resource.release() with my_context(...) as res: play_with(res)
6. async context manager § contextlib.contextmanager의 async 버전 • 이미 3rd-party asyncio_extras, asyncio-contextmanager 패키지 존재 • Python 3.7에서 contextlib.asynccontextmanager 공식 지원 예정 (bpo-29679) @contextlib.contextmanager def mycontext(): some_init() yield some_shutdown() @aiotools.actxmanager async def mycontext(): await some_init() yield await some_shutdown()
7. async context group § contextlib.ExitStack – 여러 개의 context manager 중첩 가능 § 그런데 각각의 context manager가 비동기라면? • 그리고 굳이 순서대로 실행될 필요가 없다면? § aiotools.AsyncExitStackAsyncContextGroup! • 내부적으로 asyncio.gather() 사용 @aiotools.actxmgr async def ctx(a): await asyncio.sleep(0.1) yield a + 10 await asyncio.sleep(0.1) ctxgrp = aiotools.actxgroup(ctx(i) for i in range(3)) async with ctxgrp as values: assert values[0] == 10 assert values[1] == 11 assert values[2] == 12
8. asyncio multi-process server § Event loop 주의사항 • 반드시 하나의 thread에 하나의 event loop만 사용할 것 • 다른 thread의 event loop를 사용하지 말 것 § aiotools.start_server • 다중 event loop worker 수명주기 관리 • threading 또는 multiprocessing (no GIL!) 활용 • SIGINT/SIGTERM 대응 처리 포함 • async context manager를 활용해서 깔끔하게 만들어보자!
9. aiotools.start_server API § worker, global main, extra process 지정 가능 • global main과 extra process는 synchronous function 사용 • worker 함수에는 loop, pidx, args 전달 def start_server(worker_actxmgr: AbstractAsyncContextManager, main_ctxmgr: Optional[AbstractContextManager]=None, extra_procs: Iterable[Callable]=tuple(), stop_signals: Iterable[signal.Signals]=( signal.SIGINT, signal.SIGTERM), num_workers: int=1, use_threading: bool=False, args: Iterable[Any]=tuple()): ...
10. aiotools.start_server API extra_proc individual event loop per worker aiotools.start_server join fork user main exit user main enter worker enter worker enter extra_proc worker exit worker exit
11. 예제 – Socket Echo / HTTP Server § 프로그램 구조 main worker worker worker worker 0.0.0.0:8888 0.0.0.0:8888 0.0.0.0:8888 0.0.0.0:8888
12. 예제 – Socket Echo Server async def echo(reader, writer): data = await reader.read(100) writer.write(data) await writer.drain() writer.close() @aiotools.actxmgr async def worker_main(loop, pidx, args): server = await asyncio.start_server( echo, '0.0.0.0', 8888, reuse_port=True, loop=loop) yield # wait until terminated server.close() await server.wait_closed() if __name__ == '__main__': aiotools.start_server( worker_main, num_workers=4)
13. 예제 – HTTP Server from aiohttp import web @aiotools.actxmgr async def worker_main(loop, pidx, args): app = web.Application() web_handler = app.make_handler() server = await loop.create_server( web_handler, host='0.0.0.0', port=8888, reuse_port=True) try: yield finally: server.close() await server.wait_closed() await app.shutdown() await web_handler.finish_connections(60.0) await app.cleanup() if __name__ == '__main__': aiotools.start_server( worker_main, num_workers=4)
14. 예제 – ZeroMQ Server § 프로그램 구조 main worker worker worker worker ipc://incoming (PULL) ipc://incoming (PULL) ipc://incoming (PULL) ipc://incoming (PULL) router ipc://incoming (PUSH) 0.0.0.0:5000 (PULL)
15. 예제 – ZeroMQ Server § Router def router_main(_, pidx, args): ctx = zmq.Context() ctx.linger = 0 in_sock = ctx.socket(zmq.PULL) in_sock.bind('tcp://*:5000') out_sock = ctx.socket(zmq.PUSH) out_sock.bind('ipc://incoming') try: zmq.proxy(in_sock, out_sock) except KeyboardInterrupt: pass except: logger.exception('unexpected error') finally: in_sock.close() out_sock.close() ctx.term() if __name__ == '__main__': server = aiotools.start_server( worker_main, num_workers=4, extra_procs=[router_main], )
16. 예제 – ZeroMQ Server § Worker async def process_incoming(router): while True: try: data = await router.read() except aiozmq.ZmqStreamClosed: break print(data) @aiotools.actxmgr async def worker_main(loop, pidx, args): router = await aiozmq.create_zmq_stream( zmq.PULL, connect='ipc://incoming') task = loop.create_task(process_incoming(router)) yield router.close() await task if __name__ == '__main__': server = aiotools.start_server( worker_main, num_workers=4, extra_procs=[router_main], )
17. async timer § 주기적으로 실행해야 하는 clean up / scrubbing / heartbeat § 옵션에 따라 interval보다 오래 실행하는 경우 자동 cancel 제공 count = 0 async def counter(interval): global count count += 1 timer = aiotools.create_timer(counter, 0.1) await asyncio.sleep(10) timer.cancel() await timer
18. aiotools에 적용한 CI 기법 § Flake8 + pytest + codecov § PyPI 자동 배포 • Travis CI의 build stage 기능과 encrypted password 활용하여 tagged commit인 경우 자동으로 wheel 패키지 빌드 및 PyPI 업로드 • README.rst + CHANGES.rst 내용을 setup.py 스크립트의 long_description에 넣어 pypi.python.org에서 프로젝트 설명이 예쁘게 잘 나오도록 함 (aiohttp 프로젝트에서 가져옴)
19. aiotools 향후 로드맵 § aiotools.start_server API 강화 • global main / extra process도 asyncio loop 선택적 제공 • per-process logging 유틸리티 제공 • SIGHUP, SIGUSR1 이용한 graceful restart/reload § 신규 기능 • AsyncBarrier § CI 개선 • uvloop와 tokio 라이브러리에 대한 on/off 테스트 추가
20. 유용한 asyncio 라이브러리들 § aioconsole $ apython Python 3.6.2 (default, Jul 18 2017, 02:39:19) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin Type "help", "copyright", "credits" or "license" for more information. --- This console is running in an asyncio event loop. It allows you to wait for coroutines using the 'await' syntax. Try: await asyncio.sleep(1, result=3, loop=loop) --- >>> import asyncio >>> await asyncio.sleep(1, result=3, loop=loop) 3 (내가 만들지는 않았지만)
21. 유용한 asyncio 라이브러리들 § aiomonitor • asyncio task를 외부에서 모니터링할 수 있게 해주는 유틸리티 web = aiohttp.web.Application() ... with aiomonitor.start_monitor(loop=loop): web.run_app(app, port=8090, host='localhost') $ nc localhost 50101 Asyncio Monitor: 1 tasks running Type help for commands monitor >>>
22. 대안 asyncio 프레임워크 § curio & trio • async/await 문법만 활용하고 asyncio를 사용하지 않는 대안 프레임워크 • 특징 : 항상 명시적인 coroutine task 관리 (spawn, join, cancel) • 최대한의 flexibility vs. 항상 정확한 semantic § tokio • Rust tokio 라이브러리에 기반한 event loop 구현 (uvloop와 사용법 동일) • 성능 : uvloop과 유사하거나 더 좋음 (ref: 원작자의 reddit 댓글) • PyO3 (Python용 Rust binding) 프로젝트와 함께 진행
23. 저와 이야기 & 코딩해요! § Q&A / BoF – 211호 (14:00 ~ 15:00) § 8월 15일 10am-6pm : aiodocker ( + aiotools) Sprint
24. References § https://docs.python.org/3/library/contextlib.html § https://github.com/agronholm/asyncio_extras § https://github.com/sashgorokhov/asyncio-contextmanager § https://bugs.python.org/issue29679 § https://github.com/dabeaz/curio § https://github.com/python-trio/trio § https://github.com/PyO3/tokio/ § https://github.com/PyO3/pyo3 § https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/ § https://forum.dabeaz.com/t/curio-and-trio-thoughts/173 § https://www.reddit.com/r/rust/comments/6aix0w/python_asyncio_event_loop_written_in_rust_xpost/