Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[PyCon KR 2017] Meet aiotools: asyncio idiom library

[PyCon KR 2017] Meet aiotools: asyncio idiom library

Describes the aiotools library – its background, key features, and code examples about how to use it. Also introduces other helper/alternative asyncio libraries.

Joongi Kim

August 13, 2017
Tweet

More Decks by Joongi Kim

Other Decks in Programming

Transcript

  1. Meet aiotools:
    asyncio idiom library
    김준기
    CTO, Lablup Inc. / 글로벌 오픈프론티어 4기

    View full-size slide

  2. github.com/achimnol

    View full-size slide

  3. asyncio
    § 이제 다 아시죠?
    § 하지만 실제로 쓰다보면...
    § Python 3.6부터 가능해진 것 — async generator
    async def fetch_data():
    async with connect_db() as conn:
    async for row in conn.fetch(query):
    yield row
    async for row in fetch_data():
    print(row)

    View full-size slide

  4. aiotools!
    § Lablup.AI 플랫폼 개발하면서 반복되는 코드를 리팩토링한 결과물
    § Python 3.6의 async generator를 잘 써보자!
    § aiohttp, aiodocker 프로젝트에서 배운 CI 기법들을 적용해보자!
    § (asyncio 프로젝트들이 핫하니 좋은 이름 빨리 선점해볼까?!)
    ⭐는 사랑입니다!
    https://github.com/achimnol/aiotools

    View full-size slide

  5. Reminder: context manager
    class MyContext:
    def __init__(self, ...):
    self.resource = Resource(...)
    def __enter__(self):
    self.resource.acquire()
    return self.resource
    def __exit__(self, exc_type, exc_val, tb):
    self.resource.release()
    @contextlib.contextmanager
    def my_context(...):
    resource = Resource(...)
    resource.acquire()
    try:
    yield resource
    finally:
    resource.release()
    with my_context(...) as res:
    play_with(res)

    View full-size slide

  6. async context manager
    § contextlib.contextmanager의 async 버전
    • 이미 3rd-party asyncio_extras, asyncio-contextmanager 패키지 존재
    • Python 3.7에서 contextlib.asynccontextmanager 공식 지원 예정
    (bpo-29679)
    @contextlib.contextmanager
    def mycontext():
    some_init()
    yield
    some_shutdown()
    @aiotools.actxmanager
    async def mycontext():
    await some_init()
    yield
    await some_shutdown()

    View full-size slide

  7. async context group
    § contextlib.ExitStack – 여러 개의 context manager 중첩 가능
    § 그런데 각각의 context manager가 비동기라면?
    • 그리고 굳이 순서대로 실행될 필요가 없다면?
    § aiotools.AsyncExitStackAsyncContextGroup!
    • 내부적으로 asyncio.gather() 사용
    @aiotools.actxmgr
    async def ctx(a):
    await asyncio.sleep(0.1)
    yield a + 10
    await asyncio.sleep(0.1)
    ctxgrp = aiotools.actxgroup(ctx(i) for i in range(3))
    async with ctxgrp as values:
    assert values[0] == 10
    assert values[1] == 11
    assert values[2] == 12

    View full-size slide

  8. asyncio multi-process server
    § Event loop 주의사항
    • 반드시 하나의 thread에 하나의 event loop만 사용할 것
    • 다른 thread의 event loop를 사용하지 말 것
    § aiotools.start_server
    • 다중 event loop worker 수명주기 관리
    • threading 또는 multiprocessing (no GIL!) 활용
    • SIGINT/SIGTERM 대응 처리 포함
    • async context manager를 활용해서 깔끔하게 만들어보자!

    View full-size slide

  9. aiotools.start_server API
    § worker, global main, extra process 지정 가능
    • global main과 extra process는 synchronous function 사용
    • worker 함수에는 loop, pidx, args 전달
    def start_server(worker_actxmgr: AbstractAsyncContextManager,
    main_ctxmgr: Optional[AbstractContextManager]=None,
    extra_procs: Iterable[Callable]=tuple(),
    stop_signals: Iterable[signal.Signals]=(
    signal.SIGINT,
    signal.SIGTERM),
    num_workers: int=1,
    use_threading: bool=False,
    args: Iterable[Any]=tuple()):
    ...

    View full-size slide

  10. aiotools.start_server API
    extra_proc
    individual
    event loop
    per worker
    aiotools.start_server
    join
    fork
    user main exit
    user main enter
    worker enter worker enter
    extra_proc
    worker exit worker exit

    View full-size slide

  11. 예제 – Socket Echo / HTTP Server
    § 프로그램 구조
    main
    worker worker worker worker
    0.0.0.0:8888 0.0.0.0:8888 0.0.0.0:8888 0.0.0.0:8888

    View full-size slide

  12. 예제 – Socket Echo Server
    async def echo(reader, writer):
    data = await reader.read(100)
    writer.write(data)
    await writer.drain()
    writer.close()
    @aiotools.actxmgr
    async def worker_main(loop, pidx, args):
    server = await asyncio.start_server(
    echo, '0.0.0.0', 8888,
    reuse_port=True, loop=loop)
    yield # wait until terminated
    server.close()
    await server.wait_closed()
    if __name__ == '__main__':
    aiotools.start_server(
    worker_main,
    num_workers=4)

    View full-size slide

  13. 예제 – HTTP Server
    from aiohttp import web
    @aiotools.actxmgr
    async def worker_main(loop, pidx, args):
    app = web.Application()
    web_handler = app.make_handler()
    server = await loop.create_server(
    web_handler,
    host='0.0.0.0', port=8888,
    reuse_port=True)
    try:
    yield
    finally:
    server.close()
    await server.wait_closed()
    await app.shutdown()
    await web_handler.finish_connections(60.0)
    await app.cleanup()
    if __name__ == '__main__':
    aiotools.start_server(
    worker_main,
    num_workers=4)

    View full-size slide

  14. 예제 – ZeroMQ Server
    § 프로그램 구조
    main
    worker worker worker worker
    ipc://incoming
    (PULL)
    ipc://incoming
    (PULL)
    ipc://incoming
    (PULL)
    ipc://incoming
    (PULL)
    router
    ipc://incoming
    (PUSH)
    0.0.0.0:5000
    (PULL)

    View full-size slide

  15. 예제 – ZeroMQ Server
    § Router
    def router_main(_, pidx, args):
    ctx = zmq.Context()
    ctx.linger = 0
    in_sock = ctx.socket(zmq.PULL)
    in_sock.bind('tcp://*:5000')
    out_sock = ctx.socket(zmq.PUSH)
    out_sock.bind('ipc://incoming')
    try:
    zmq.proxy(in_sock, out_sock)
    except KeyboardInterrupt:
    pass
    except:
    logger.exception('unexpected error')
    finally:
    in_sock.close()
    out_sock.close()
    ctx.term()
    if __name__ == '__main__':
    server = aiotools.start_server(
    worker_main,
    num_workers=4,
    extra_procs=[router_main],
    )

    View full-size slide

  16. 예제 – ZeroMQ Server
    § Worker
    async def process_incoming(router):
    while True:
    try:
    data = await router.read()
    except aiozmq.ZmqStreamClosed:
    break
    print(data)
    @aiotools.actxmgr
    async def worker_main(loop, pidx, args):
    router = await aiozmq.create_zmq_stream(
    zmq.PULL, connect='ipc://incoming')
    task = loop.create_task(process_incoming(router))
    yield
    router.close()
    await task
    if __name__ == '__main__':
    server = aiotools.start_server(
    worker_main,
    num_workers=4,
    extra_procs=[router_main],
    )

    View full-size slide

  17. async timer
    § 주기적으로 실행해야 하는 clean up / scrubbing / heartbeat
    § 옵션에 따라 interval보다 오래 실행하는 경우 자동 cancel 제공
    count = 0
    async def counter(interval):
    global count
    count += 1
    timer = aiotools.create_timer(counter, 0.1)
    await asyncio.sleep(10)
    timer.cancel()
    await timer

    View full-size slide

  18. aiotools에 적용한 CI 기법
    § Flake8 + pytest + codecov
    § PyPI 자동 배포
    • Travis CI의 build stage 기능과 encrypted password 활용하여 tagged
    commit인 경우 자동으로 wheel 패키지 빌드 및 PyPI 업로드
    • README.rst + CHANGES.rst 내용을 setup.py 스크립트의
    long_description에 넣어 pypi.python.org에서 프로젝트 설명이 예쁘게 잘
    나오도록 함 (aiohttp 프로젝트에서 가져옴)

    View full-size slide

  19. aiotools 향후 로드맵
    § aiotools.start_server API 강화
    • global main / extra process도 asyncio loop 선택적 제공
    • per-process logging 유틸리티 제공
    • SIGHUP, SIGUSR1 이용한 graceful restart/reload
    § 신규 기능
    • AsyncBarrier
    § CI 개선
    • uvloop와 tokio 라이브러리에 대한 on/off 테스트 추가

    View full-size slide

  20. 유용한 asyncio 라이브러리들
    § aioconsole
    $ apython
    Python 3.6.2 (default, Jul 18 2017, 02:39:19)
    [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
    Type "help", "copyright", "credits" or "license" for more
    information.
    ---
    This console is running in an asyncio event loop.
    It allows you to wait for coroutines using the 'await' syntax.
    Try: await asyncio.sleep(1, result=3, loop=loop)
    ---
    >>> import asyncio
    >>> await asyncio.sleep(1, result=3, loop=loop)
    3
    (내가 만들지는 않았지만)

    View full-size slide

  21. 유용한 asyncio 라이브러리들
    § aiomonitor
    • asyncio task를 외부에서 모니터링할 수 있게 해주는 유틸리티
    web = aiohttp.web.Application()
    ...
    with aiomonitor.start_monitor(loop=loop):
    web.run_app(app, port=8090, host='localhost')
    $ nc localhost 50101
    Asyncio Monitor: 1 tasks running
    Type help for commands monitor
    >>>

    View full-size slide

  22. 대안 asyncio 프레임워크
    § curio & trio
    • async/await 문법만 활용하고 asyncio를 사용하지 않는 대안 프레임워크
    • 특징 : 항상 명시적인 coroutine task 관리 (spawn, join, cancel)
    • 최대한의 flexibility vs. 항상 정확한 semantic
    § tokio
    • Rust tokio 라이브러리에 기반한 event loop 구현 (uvloop와 사용법 동일)
    • 성능 : uvloop과 유사하거나 더 좋음 (ref: 원작자의 reddit 댓글)
    • PyO3 (Python용 Rust binding) 프로젝트와 함께 진행

    View full-size slide

  23. 저와 이야기 & 코딩해요!
    § Q&A / BoF – 211호 (14:00 ~ 15:00)
    § 8월 15일 10am-6pm : aiodocker ( + aiotools) Sprint

    View full-size slide

  24. References
    § https://docs.python.org/3/library/contextlib.html
    § https://github.com/agronholm/asyncio_extras
    § https://github.com/sashgorokhov/asyncio-contextmanager
    § https://bugs.python.org/issue29679
    § https://github.com/dabeaz/curio
    § https://github.com/python-trio/trio
    § https://github.com/PyO3/tokio/
    § https://github.com/PyO3/pyo3
    § https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/
    § https://forum.dabeaz.com/t/curio-and-trio-thoughts/173
    § https://www.reddit.com/r/rust/comments/6aix0w/python_asyncio_event_loop_written_in_rust_xpost/

    View full-size slide