[PyCon KR 2017] Meet aiotools: asyncio idiom library

Describes the aiotools library – its background, key features, and code examples about how to use it. Also introduces other helper/alternative asyncio libraries.

Transcript

1. Meet aiotools:
   asyncio idiom library
   김준기
   CTO, Lablup Inc. / 글로벌 오픈프론티어 4기
   

   View Slide

2. github.com/achimnol
   

   View Slide

3. asyncio
   § 이제 다 아시죠?
   § 하지만 실제로 쓰다보면...
   § Python 3.6부터 가능해진 것 — async generator
   async def fetch_data():
   async with connect_db() as conn:
   async for row in conn.fetch(query):
   yield row
   async for row in fetch_data():
   print(row)
   

   View Slide

4. aiotools!
   § Lablup.AI 플랫폼 개발하면서 반복되는 코드를 리팩토링한 결과물
   § Python 3.6의 async generator를 잘 써보자!
   § aiohttp, aiodocker 프로젝트에서 배운 CI 기법들을 적용해보자!
   § (asyncio 프로젝트들이 핫하니 좋은 이름 빨리 선점해볼까?!)
   ⭐는 사랑입니다!
   https://github.com/achimnol/aiotools
   

   View Slide

5. Reminder: context manager
   class MyContext:
   def __init__(self, ...):
   self.resource = Resource(...)
   def __enter__(self):
   self.resource.acquire()
   return self.resource
   def __exit__(self, exc_type, exc_val, tb):
   self.resource.release()
   @contextlib.contextmanager
   def my_context(...):
   resource = Resource(...)
   resource.acquire()
   try:
   yield resource
   finally:
   resource.release()
   with my_context(...) as res:
   play_with(res)
   

   View Slide

6. async context manager
   § contextlib.contextmanager의 async 버전
   • 이미 3rd-party asyncio_extras, asyncio-contextmanager 패키지 존재
   • Python 3.7에서 contextlib.asynccontextmanager 공식 지원 예정
   (bpo-29679)
   @contextlib.contextmanager
   def mycontext():
   some_init()
   yield
   some_shutdown()
   @aiotools.actxmanager
   async def mycontext():
   await some_init()
   yield
   await some_shutdown()
   

   View Slide

7. async context group
   § contextlib.ExitStack – 여러 개의 context manager 중첩 가능
   § 그런데 각각의 context manager가 비동기라면?
   • 그리고 굳이 순서대로 실행될 필요가 없다면?
   § aiotools.AsyncExitStackAsyncContextGroup!
   • 내부적으로 asyncio.gather() 사용
   @aiotools.actxmgr
   async def ctx(a):
   await asyncio.sleep(0.1)
   yield a + 10
   await asyncio.sleep(0.1)
   ctxgrp = aiotools.actxgroup(ctx(i) for i in range(3))
   async with ctxgrp as values:
   assert values[0] == 10
   assert values[1] == 11
   assert values[2] == 12
   

   View Slide

8. asyncio multi-process server
   § Event loop 주의사항
   • 반드시 하나의 thread에 하나의 event loop만 사용할 것
   • 다른 thread의 event loop를 사용하지 말 것
   § aiotools.start_server
   • 다중 event loop worker 수명주기 관리
   • threading 또는 multiprocessing (no GIL!) 활용
   • SIGINT/SIGTERM 대응 처리 포함
   • async context manager를 활용해서 깔끔하게 만들어보자!
   

   View Slide

9. aiotools.start_server API
   § worker, global main, extra process 지정 가능
   • global main과 extra process는 synchronous function 사용
   • worker 함수에는 loop, pidx, args 전달
   def start_server(worker_actxmgr: AbstractAsyncContextManager,
   main_ctxmgr: Optional[AbstractContextManager]=None,
   extra_procs: Iterable[Callable]=tuple(),
   stop_signals: Iterable[signal.Signals]=(
   signal.SIGINT,
   signal.SIGTERM),
   num_workers: int=1,
   use_threading: bool=False,
   args: Iterable[Any]=tuple()):
   ...
   

   View Slide

10. aiotools.start_server API
    extra_proc
    individual
    event loop
    per worker
    aiotools.start_server
    join
    fork
    user main exit
    user main enter
    worker enter worker enter
    extra_proc
    worker exit worker exit
    

    View Slide

11. 예제 – Socket Echo / HTTP Server
    § 프로그램 구조
    main
    worker worker worker worker
    0.0.0.0:8888 0.0.0.0:8888 0.0.0.0:8888 0.0.0.0:8888
    

    View Slide

12. 예제 – Socket Echo Server
    async def echo(reader, writer):
    data = await reader.read(100)
    writer.write(data)
    await writer.drain()
    writer.close()
    @aiotools.actxmgr
    async def worker_main(loop, pidx, args):
    server = await asyncio.start_server(
    echo, '0.0.0.0', 8888,
    reuse_port=True, loop=loop)
    yield # wait until terminated
    server.close()
    await server.wait_closed()
    if __name__ == '__main__':
    aiotools.start_server(
    worker_main,
    num_workers=4)
    

    View Slide

13. 예제 – HTTP Server
    from aiohttp import web
    @aiotools.actxmgr
    async def worker_main(loop, pidx, args):
    app = web.Application()
    web_handler = app.make_handler()
    server = await loop.create_server(
    web_handler,
    host='0.0.0.0', port=8888,
    reuse_port=True)
    try:
    yield
    finally:
    server.close()
    await server.wait_closed()
    await app.shutdown()
    await web_handler.finish_connections(60.0)
    await app.cleanup()
    if __name__ == '__main__':
    aiotools.start_server(
    worker_main,
    num_workers=4)
    

    View Slide

14. 예제 – ZeroMQ Server
    § 프로그램 구조
    main
    worker worker worker worker
    ipc://incoming
    (PULL)
    ipc://incoming
    (PULL)
    ipc://incoming
    (PULL)
    ipc://incoming
    (PULL)
    router
    ipc://incoming
    (PUSH)
    0.0.0.0:5000
    (PULL)
    

    View Slide

15. 예제 – ZeroMQ Server
    § Router
    def router_main(_, pidx, args):
    ctx = zmq.Context()
    ctx.linger = 0
    in_sock = ctx.socket(zmq.PULL)
    in_sock.bind('tcp://*:5000')
    out_sock = ctx.socket(zmq.PUSH)
    out_sock.bind('ipc://incoming')
    try:
    zmq.proxy(in_sock, out_sock)
    except KeyboardInterrupt:
    pass
    except:
    logger.exception('unexpected error')
    finally:
    in_sock.close()
    out_sock.close()
    ctx.term()
    if __name__ == '__main__':
    server = aiotools.start_server(
    worker_main,
    num_workers=4,
    extra_procs=[router_main],
    )
    

    View Slide

16. 예제 – ZeroMQ Server
    § Worker
    async def process_incoming(router):
    while True:
    try:
    data = await router.read()
    except aiozmq.ZmqStreamClosed:
    break
    print(data)
    @aiotools.actxmgr
    async def worker_main(loop, pidx, args):
    router = await aiozmq.create_zmq_stream(
    zmq.PULL, connect='ipc://incoming')
    task = loop.create_task(process_incoming(router))
    yield
    router.close()
    await task
    if __name__ == '__main__':
    server = aiotools.start_server(
    worker_main,
    num_workers=4,
    extra_procs=[router_main],
    )
    

    View Slide

17. async timer
    § 주기적으로 실행해야 하는 clean up / scrubbing / heartbeat
    § 옵션에 따라 interval보다 오래 실행하는 경우 자동 cancel 제공
    count = 0
    async def counter(interval):
    global count
    count += 1
    timer = aiotools.create_timer(counter, 0.1)
    await asyncio.sleep(10)
    timer.cancel()
    await timer
    

    View Slide

18. aiotools에 적용한 CI 기법
    § Flake8 + pytest + codecov
    § PyPI 자동 배포
    • Travis CI의 build stage 기능과 encrypted password 활용하여 tagged
    commit인 경우 자동으로 wheel 패키지 빌드 및 PyPI 업로드
    • README.rst + CHANGES.rst 내용을 setup.py 스크립트의
    long_description에 넣어 pypi.python.org에서 프로젝트 설명이 예쁘게 잘
    나오도록 함 (aiohttp 프로젝트에서 가져옴)
    

    View Slide

19. aiotools 향후 로드맵
    § aiotools.start_server API 강화
    • global main / extra process도 asyncio loop 선택적 제공
    • per-process logging 유틸리티 제공
    • SIGHUP, SIGUSR1 이용한 graceful restart/reload
    § 신규 기능
    • AsyncBarrier
    § CI 개선
    • uvloop와 tokio 라이브러리에 대한 on/off 테스트 추가
    

    View Slide

20. 유용한 asyncio 라이브러리들
    § aioconsole
    $ apython
    Python 3.6.2 (default, Jul 18 2017, 02:39:19)
    [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
    Type "help", "copyright", "credits" or "license" for more
    information.
    ---
    This console is running in an asyncio event loop.
    It allows you to wait for coroutines using the 'await' syntax.
    Try: await asyncio.sleep(1, result=3, loop=loop)
    ---
    >>> import asyncio
    >>> await asyncio.sleep(1, result=3, loop=loop)
    3
    (내가 만들지는 않았지만)
    

    View Slide

21. 유용한 asyncio 라이브러리들
    § aiomonitor
    • asyncio task를 외부에서 모니터링할 수 있게 해주는 유틸리티
    web = aiohttp.web.Application()
    ...
    with aiomonitor.start_monitor(loop=loop):
    web.run_app(app, port=8090, host='localhost')
    $ nc localhost 50101
    Asyncio Monitor: 1 tasks running
    Type help for commands monitor
    >>>
    

    View Slide

22. 대안 asyncio 프레임워크
    § curio & trio
    • async/await 문법만 활용하고 asyncio를 사용하지 않는 대안 프레임워크
    • 특징 : 항상 명시적인 coroutine task 관리 (spawn, join, cancel)
    • 최대한의 flexibility vs. 항상 정확한 semantic
    § tokio
    • Rust tokio 라이브러리에 기반한 event loop 구현 (uvloop와 사용법 동일)
    • 성능 : uvloop과 유사하거나 더 좋음 (ref: 원작자의 reddit 댓글)
    • PyO3 (Python용 Rust binding) 프로젝트와 함께 진행
    

    View Slide

23. 저와 이야기 & 코딩해요!
    § Q&A / BoF – 211호 (14:00 ~ 15:00)
    § 8월 15일 10am-6pm : aiodocker ( + aiotools) Sprint
    

    View Slide

24. References
    § https://docs.python.org/3/library/contextlib.html
    § https://github.com/agronholm/asyncio_extras
    § https://github.com/sashgorokhov/asyncio-contextmanager
    § https://bugs.python.org/issue29679
    § https://github.com/dabeaz/curio
    § https://github.com/python-trio/trio
    § https://github.com/PyO3/tokio/
    § https://github.com/PyO3/pyo3
    § https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/
    § https://forum.dabeaz.com/t/curio-and-trio-thoughts/173
    § https://www.reddit.com/r/rust/comments/6aix0w/python_asyncio_event_loop_written_in_rust_xpost/
    

    View Slide